acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Population Prior: Reliable and Amortizable Person Representations in Models of Human Decision-Making

Abstract

Most learned models of human decision-making represent a population, not a person. We ask whether a person's departures from such a model persist, improve prediction, and can be inferred quickly. Chess is our testbed: 2.6 million real-game decisions at positions whose difficulty other people rated; person fits use 545 players with at least one hundred each. A model over a frozen population reference infers each person's tempo, dispersion and ability beyond rating. They replicate across date-split halves (length-corrected tempo r = 0.79); a year apart agreement is lower; at equal decision counts, tempo and ability show no detectable drop. Per decision the person adds little: the held-out think-time gain, 0.010 nats, misses our 0.02-nat margin; in-game clock history beats static person offsets on timing; ability beyond rating adds accuracy, below margin. An amortized encoder infers held-out players from few decisions, 0.9% short of a fit on all their decisions against a 0.5% margin. Instrumented by the time control, extra time's estimated effect on accuracy is positive for players whose think time it changes, opposite to the within-person association (exclusion restriction checked, not proved). The traits are reliable; their value depends on the baseline; timing is partly amortizable, ability not yet.

Primary area:
applications to neuroscience & cognitive science
Keywords:
person representations, amortized inference, hierarchical model, split-half reliability, response time, calibrated items, instrumental variables, human-decision-making, chess

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_3jN2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies models of human decision-making aimed at capturing the ways in which one individual deviates from the aggregate model defined over the whole population. It asks whether these individual-level models will be more effective at predicting an individual's actions, relative to the overall model. It also considers a set of temporal questions, including how stable the individual-level model of a person's decisions are over time, and how the amount of time the person has to make a decision in an instance-by-instance sense affects their decision-making.

Critical strengths

Training a per-person model is an important problem that is relevant to many decision-making contexts. In settings such as chess where there is rich temporal data, using the timing information is an important potential signal.

Critical weaknesses

Despite a subsection in the Introduction entitled "Contributions" with a numbered list, the paper has difficulties making what its nove contributions actually are relative to the existing body of literature that builds individual-level models and incorporates timing information. The overview of the contributions in the subsection in the Introduction entitled "Contributions" begins with a summary sentence that doesn't convey what the paper actually accomplishes: "The finding is a tension: measurable personal differences persist even where their added value per decision is small, and against the game’s own clock history a static person term adds only a small accuracy signal."

Very little of the main body of the paper provides insight into the comparison with the large body of related work introduced in Section 2. If Table 1 at the bottom of page 7 is supposed to be an important place for seeing the benefits of the approach in the current submission, it needs to be expanded upon and explained much more clearly.

All of these attempts at explaining the contributions are hampered by an extremely compressed writing style that doesn't surface the key new points that the current paper brings to the problem.

Questions for authors

Since the paper does not succeed in providing a succinct description of its contributions, could you answer the following two questions: (1) What is the specific method by which you are evaluating your approach relative to the approaches in past work? (2) What is the concrete evidence via this method that your approach represents an improvement over existing approaches to the problem?

1 comment
AuthorsOpenReview ↗

what is new, the evidence, and the comparison with prior work

The paper does not claim a better person model. It combines standard tests with two controls and a known-trait ceiling to assess what a person term adds, and shows cases where they change a reading.

Your first question: the method for evaluating against past work. For chess timing and solving, every comparator is scored on the same held-out decisions, against the same frozen population model, with paired player-level intervals. Earlier approaches enter as rows of Table 1: population models (Maia-2 through our reference; Allie on its blitz subset, calibrated on this cohort), simple per-person models (a tuned tempo offset; a joint speed–ability model, post hoc) and models that read recent history (this game's clock record; the person's recent solves). Table 55 (Appendix Y.11) names the analysis that stands for each line of prior work. For moves we use our own adaptations of Maia-2 and Maia-3 (Table 2); published per-player models enter only through their reported numbers.

Your second question: concrete evidence of improvement. Over the population model: yes, but small. The person vector improves think-time prediction by 0.0134 nats per decision over our Maia-2-based reference, below our 0.02 margin (Table 1). Over existing person models: no. A tuned one-number tempo offset is not detectably worse (difference 0.0030 nats [−0.0012, 0.0072]; exploratory). What the paper adds is the second test's two controls, and three cases show why they matter.

(i) A published comparison. From about twenty games per player, Maia4All reports top-1 accuracy of 53.22% against 51.46% for Maia-2, and 51.89% for a strength-only variant (110 players, ten above 2000; table on page 3). The strength-only variant is itself fitted on each player's games. That comparison has no arm fitted without the player's data, so its gain cannot be split into a part that needs the player's data and a part that does not. Our second test asks for that arm.

(ii) Our move data (Table 2; exploratory). For new blitz players rated 2000 or above, a correction to Maia-2 with no per-player parameters, fitted on other players' games, gives 0.74 of what our per-player embedding gains. For players rated 1100 to 1499 it gives 0.11. The answer also depends on the population model: one correction recipe gains 0.0176 nats per move on Maia-2 and 0.0015 on the newer Maia-3 (post hoc; Maia-3's training period covers these months). Our adaptations use about a hundred games per player and are no test of published per-player models.

(iii) The tests can say yes (Table 3; exploratory). In three of five further data sets, a person term clears the held-out margins against a correction fitted without the person; in a fourth, a tutoring log, it clears them over the reference (Table 3, note c). It does not against the person's recent record, where one exists.

Novelty against prior individual-level models. Each test alone is standard, and Section 2 credits its sources. New is their joint use: two controls and a ceiling, the gain a perfectly known trait would give. The ceiling yields a finding of its own. In chess the gains reach about two-thirds (tempo) and 76% (ability) of what a perfectly known static trait would give, and a known ability would still fall short of its margin (Section 5.2). The controls also changed our own readings: in Table 4, four of our plain comparisons shrank or vanished under a control (exploratory).

The summary sentence of the contributions. We agree, and have removed that sentence. The Contributions paragraph now opens: "We give tests that detect when a gain over a population model overstates the value of a person term, and apply them to six data sets."

Comparison with related work in the main body. A table on page 3 now maps each line of prior work to the analysis here that tests it, with its result. Sections 5.2 and 5.5 then make the comparisons: per-person offsets and history-aware models for timing; Maia-2, Maia-3 and Maia4All's published comparison for moves.

Table 1. Table 1 now reads positive as better. Its first number column is each comparator's timing gain over the reference. The next two give what the person vector adds beyond that comparator, for think time and for solving. For rows marked †, it is what the vector adds when refitted on top of the comparator. Section 5.2 walks through the rows.

Compressed writing. We agree. The abstract now opens with the question, each results subsection (5.1 to 5.5) opens with its answer, and most numbers of Section 5.5 sit in Tables 2 to 4.

Thank you for the detailed, thoughtful review. We reply late because we revised the paper thoroughly rather than answer in haste. Every point you raised is taken up, and the paper is revised throughout. We would be grateful if you read the revised version.

Reviewer_4J7eOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper attempts to improve personalized modelling in chess. The authors construct a dataset of chess puzzles and the original games they came from. They use this dataset to build a population model by training gradient boosted predictors on a small number of extracted Maia 2 features and context variables (e.g. clock time). They then add four personalized parameters per player: tempo, dispersion, ability beyond rating, and instant move probability. They find that tempo, ability, and dispersion are reliable, with tempo in particular remaining reliable even one year later. They then use an encoder to predict player parameter values from a small number of observations, finding good results for tempo. The paper also replicates its analysis on Maia3. Ultimately, it contributes a simple framework for estimating player time use propensities and a mixed set of positive and negative results about the predictability of those propensities.

Critical strengths
  1. the dataset is cleverly constructed: using puzzles and the games they originally came from is a nice way to measure player behaviour in the wild while accessing an external measure of position difficulty
  2. pre-specified significance margins are appreciated
  3. the result that players exhibit stable tempo, even after a year, is cool
Critical weaknesses
  1. very niche topic; low impact
  2. does not improve upon previous methods for actually predicting individual players' moves
  3. difficult to parse LLM-like writing
  4. the choice to predict specific features of time use (e.g. dispersion) does not seem particularly well motivated or in keeping with previous chess deep learning research
  5. the paper is a statistical modelling paper first and foremost, with only a secondary deep learning component
Questions for authors
  1. Your discussion and conclusion sections seem to only be aimed at summarizing your results and their limitations. What do you see as the contribution of your work? What does it add to the machine learning literature?
  2. do you think you could get better performance with a higher dimensional player representation?
1 comment
AuthorsOpenReview ↗

what the work adds, and what we found on moves

We agree with two of your points: the paper offers no new model of individual players' moves, and its trained encoder does not beat a classical refit. What it adds is two controls that test whether a per-user gain is personal.

Your first question: the contribution, and what it adds to machine learning. Before crediting a gain over a population model such as Maia-2 to personalisation, compare it with two controls: a correction fitted without the player, and the player's recent record. Section 6 now opens with this advice. Two exploratory results on moves show why it matters (Table 2; Appendix Y.4). They come from our own adaptations of frozen Maia-2 and Maia-3 and are no test of published per-player models.

  1. On Maia-2, much of our embedding's gain needs no data from the player; how much depends on rating. We fitted on one month of blitz games from 879 new players and scored on the next. A correction with no per-player parameters gives 0.74 of what our per-player embedding gains at 2000 or above, and 0.11 at 1100 to 1499. (Maia-2's rapid checkpoint; against its blitz checkpoint the correction adds 0.0078 nats at 2000 or above, not 0.0108.)
  2. How much depends on the population model. We scored the newer Maia-3 on the paper's cohort, without move history and with one seed; its training period covers our months. One correction recipe gains 0.0015 nats per move there, against 0.0176 on Maia-2 (post hoc). As the recent record we used the current game's move history, not the player's earlier games. It improves Maia-3 by 0.068 nats per move, and our embedding still adds about 0.005 beyond it (post hoc).

Our checks cannot say why Maia-2 needs the correction: its saturated top rating bin, the selection of moves, or both.

No improvement over previous move-prediction methods. Conceded. Maia4All reports 53.22% top-1 against 51.46% for Maia-2 from about twenty games (51.89% for a strength-only variant), on 110 players of whom ten are rated above 2000, and +1.90 points above 2000 with about 2,500 games each. Our adaptations use about a hundred games, so the figures are not comparable. That comparison has no arm fitted without the player's data; our second test asks for one (table on page 3).

The margins you appreciated are smallest material gains, not significance thresholds. Those of the three chess questions were fixed before the first fit; the third test's metric was adapted later (Appendix F).

"Very niche topic". The same tests run on five further data sets: programming contests, toxicity judgments, film ratings, a tutoring log and laboratory response times (Table 3; exploratory, with the chess margins unchanged). Where the population model knows little or nothing about the person, a person term clears the held-out margins against the correction fitted without the person. For toxicity judgments that is 4 AUROC points against a 1-point margin. It does not clear them against the recent record, where one exists. We tested no downstream use.

Writing. The main text is rewritten in plain words, and each results subsection (5.1 to 5.5) opens with its answer. The AI-use statement is on page 11.

Why these features of time use. Think time is already a prediction target in chess deep learning (Allie's think-time head). Our model of log think time has a location and a scale per player. Tempo and dispersion are these two, measured relative to the reference (Section 4.1). Dispersion is reliable, but its predictive value is not established; its measured role is calibrating each player's prediction intervals (exploratory). The learned alternative is tested too: beside clock history, learned person terms of up to 32 dimensions beat our person vector by at most 0.0006 nats (exploratory; Appendix Y.5).

Statistical modelling first. We agree: this is an evaluation paper, and the encoder brings no edge (Section 5.3). The tests apply to learned representations too: the paper applies them to move embeddings and learned timing latents, and the first control to per-player adapters.

Your second question: a higher-dimensional player representation. For moves: adapters yes, a larger embedding no (exploratory). A 64-number embedding did 0.0011 nats per move worse than the correction (one seed, one regulariser). Per-player adapters gain 0.0130 over it, and 0.0093 beyond one adapter of equal capacity fitted on other players, which alone gains about 0.016 (post hoc; one seed; 100 players; Table 2, note). For timing, no (the 0.0006 nats above).

Thank you for the detailed review and for two direct questions. We reply late because we revised the paper thoroughly rather than answer in haste. Every point you raised is taken up, and the paper is revised throughout. We would be grateful if you read the revised version.

Reviewer_JpiwOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper seem to intend to address the question on how much individual behavior deviates from the population prior, and whether there can be a model that infers individual decision making based on the person’s deviation from population prior.

Critical strengths

While the question is interesting to ask. I am really confused about how the paper is logically intact...

Critical weaknesses

It is unclear to me what is the contribution of this paper, why is asking the question interesting, and how does the method of the paper proposed contribute to answering the problem.

I find this paper very difficult to follow.

Questions for authors

line 93: “the finding is a tension”?

1 comment
AuthorsOpenReview ↗

the question, why it matters, and how the method answers it

You found the question interesting but the paper unclear: what it contributes, why the question matters, and how the method answers it. Here is the paper in plain words.

The question. Models that predict human decisions describe a typical person. Many systems then add a few numbers for each individual, which we call a person term. Is that worth doing?

Why the question is interesting. When such a model predicts better, the gain is usually credited to the person term. That can mislead in two ways (Figure 1a). Part of the gain can come from fixing the typical-person model's own errors, which needs no data from the person. Part can be had with no stored person term, by a model that reads the person's recent decisions. So someone who adds a few numbers per user should first compare with two controls: a correction fitted without that user, and the user's recent record. The same choice arises for language-model agents that simulate one person, for models of each student's knowledge and for models of each annotator's labels (Section 1).

The contribution in one sentence. Tests that detect when a gain over a model of the typical person overstates the value of a per-person term, applied to chess and five other data sets (Section 1).

How the method answers it. Our fixed chess model predicts how long a typical player of a given rating thinks, and whether they find the best move. We fit four numbers per player on top of it (Section 4). Then we run three checks (the paper's tests T1 to T3):

  1. Do the personal differences hold up when measured twice?
  2. Do they improve predictions of later decisions beyond the two controls?
  3. Can a network infer them from a few decisions?

The table at the top of page 4 pairs each check with what is fitted and what it is compared against.

The logic of the paper. Each check has its own pass mark (the paper's margin): the smallest result we count as worthwhile. The pass marks were fixed for chess before the first fit. In chess the first check passes; the second and third do not. One pass mark, for solving, could not have been reached even with perfect knowledge of a player's ability. We computed that only afterwards, and Section 6 now recommends setting each pass mark from what perfect knowledge would gain.

What we find. In chess the personal differences are reliable across halves of a player's record. Yet they barely improve the prediction of a single decision. A single number for each player's pace (a tuned offset) is not detectably worse at predicting think time (Table 1; exploratory, meaning it was not among the tests fixed in advance). The first control also changes the reading of fast (blitz) chess moves. A correction fitted without the player gives most of what our per-player embedding for moves gains for strong players (rated 2000 or above), and little for weaker ones (1100 to 1499) (Table 2; exploratory). The checks can also say yes. For toxicity judgments (people rating whether online comments are toxic), the model knows only each person's demographic group. There the differences are reliable, and against a correction fitted without the person they improve held-out predictions by about four times the pass mark on one accuracy measure (AUROC). These data have no dates, so the recent-record comparison cannot be run (Table 3; exploratory).

"The finding is a tension". That sentence is gone. It meant that a difference can be measured reliably over many decisions and still improve the prediction of any one decision only slightly. Think of a slightly biased coin: many flips measure the bias precisely, yet knowing it barely helps predict the next flip. A player's pace is like that.

Hard to follow. We agree, and rewrote the main text for this. The abstract opens with the question and defines a person term, Figure 1a shows where a gain can come from, and each results subsection (5.1 to 5.5) opens with its answer.

Thank you for reading the paper carefully and for the candid feedback. We reply late because we revised the paper thoroughly rather than answer in haste. Every point you raised is taken up, and the paper is revised throughout. We would be grateful if you read the revised version.

Comments on the paper

2 comments
AuthorsOpenReview ↗

(1 of 2) Summary of the revision: the claim, what is unchanged, and what changed for each review point

We thank the three reviewers. All three found the paper hard to follow; making it readable was our first aim.

The claim. When a model with a few parameters per person predicts better than a population model, the gain is usually read as the value of personalisation. That reading can mislead. Part of the gain can repair the population model's own errors and needs no data from the person. Part can repeat what the person's recent decisions already show. The paper gives three tests of a person term: is it reliable, does it improve held-out prediction beyond those two alternatives, and can it be inferred from a few decisions?

What is unchanged. The title is new; the chess testbed, the three tests, their margins and their verdicts are those of the submission. Four comparative conclusions changed (see our comment "Corrections since submission").

What changed. (1) The main text states the thesis first, in plain words. The abstract opens with the question, Figure 1a shows where a gain can come from, and each results subsection opens with its answer. Secondary detail is in Appendix Y.14. (2) The second test now names its two controls, a correction fitted without the person and the person's recent record. The first is applied to chess moves by rating band (Table 2), and both, the second where a record exists, to five further data sets, where the tests can also say yes (Table 3); both analyses are exploratory and new since the submission. (3) Numbers corrected since submission are listed by cause in the corrections comment. To make room, the instrumented effect of think time (submitted Section 5.4) moved to Appendix P.1. Appendix Y is new.

The table maps each review point to what changed and where. Pointers are to the revised PDF.

Review pointWhat changedWhere
3jN2: what is new against prior work; method of comparisonContributions stated around one thesis; the second test names its two controls and its ceilingAbstract; Section 1
3jN2: evidence of improvementAdded exploratory evidence that the controls change a reading: moves by band, further data sets, four of our own analysesTables 2, 3 and 4
3jN2: the contributions' summary sentence; Jpiw: "the finding is a tension"That sentence removed; the Contributions paragraph opens with what the paper doesSection 1
3jN2: comparison with related workA table maps each line of prior work to the analysis that tests it and its resultTable of prior work, page 3; Appendix Y.11
3jN2: Table 1Positive is better in every column; header fixed; the text walks through the rowsTable 1; Section 5.2
3jN2, 4J7e, Jpiw: writingPlain wording; question and thesis first; three contributions; each results subsection opens with its answer; numbers in tablesAbstract; Sections 1 to 7
4J7e: "very niche topic"The same tests on five further data sets, with the limits of eachTable 3; Appendices Y.8 and Y.9
4J7e: no improvement in predicting players' movesConceded. What a correction fitted without the player gives, by rating band and on Maia-3Table 2; Appendix Y.4
4J7e: a higher-dimensional representationLarger move embeddings, per-player adapters, learned timing latentsAppendices Y.4 and Y.5
4J7e: motivation for the time-use featuresTempo and dispersion motivated as the location and scale of think timesSection 4.1
4J7e: statistical modelling firstConceded for the encoder. The same tests are applied to learned person terms (move embeddings, timing latents), and the first control to per-player adaptersSection 5.3; Table 2; Appendix Y.5
4J7e: what the work addsThe Discussion leads with the take-home: report both controls beside any personalised modelSection 6
Jpiw: the logic; how the method answers the questionEach test paired with what is fitted and what it is compared against; a simulation with a planted answerFigure 1; the table at the top of page 4; Appendix Y.10
Jpiw: the contribution; why the question is interestingThe question and why it matters, stated before any number, with a figureAbstract; Section 1; Figure 1a

How it was checked. Only the three chess questions and their margins were fixed before the first fit (the third test's metric and comparator were adapted later; Appendix F). Everything else is exploratory. Appendix Y.13 lists each rule written before an outcome, and whether it was met. Most new analyses in Tables 2 and 3 were re-computed independently with separately written code; each appendix section says which were and which were not.

AuthorsOpenReview ↗

(2 of 2) Corrections since submission: what changed in the numbers, and why

What changed in the numbers since submission, and why. No test verdict changes; four comparative conclusions do:

  1. The zero-probe control (the class mean) now beats the reference (Appendix J).
  2. A point refit now beats the reference from the first decision; before, it was worse up to three.
  3. The encoder no longer beats a marginalised classical refit; the two tie at twenty decisions.
  4. We withdraw the claim that person offsets added unchanged to the clock-history model worsen it.

Causes

  • Class-mean estimator. Random-effects means replace precision-weighted ones, which let a few players with collapsed standard errors dominate some classes (conclusion 4). The timing gain per decision is now 0.0134 nats, still below the 0.02 margin.
  • Few-shot pipeline. Corrected class means, an encoder input that was known only after the decision, population terms refitted without held-out players, a fixed training budget and more bootstrap draws changed together (conclusions 1 to 3). The shortfall from the full fit is now 1.01%, still above the 0.5% margin.
  • One input of the population model. Maia-2's probability of the solution move was masked to legal moves for solved decisions only. Defined consistently, it lowers the reference's AUROC on Table 1 decisions from 0.787 to 0.784. Neither this nor refitting population terms on the fit months changes any contrast materially (Appendix Y.2).
  • Simulation family. The submitted simulation used Johnson's SU family, not the sinh-arcsinh family it named. Appendix I reports the re-run with the named family; its tables changed and we withdrew one claim about a floor.
  • Statements. The encoder's cost comparison had timed an older refit; it saves no compute at this scale. We corrected a known-tempo benchmark, a calibration claim, the pairing of reliability intervals, several "not tested" statements and the Ethics statement on usernames. We also narrowed what was fixed in advance: the three questions and their margins, where the submission said the whole analysis plan.

The full ledger, quoting each changed sentence and number as submitted, is available on request.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.