- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies models of human decision-making aimed at capturing the ways in which one individual deviates from the aggregate model defined over the whole population. It asks whether these individual-level models will be more effective at predicting an individual's actions, relative to the overall model. It also considers a set of temporal questions, including how stable the individual-level model of a person's decisions are over time, and how the amount of time the person has to make a decision in an instance-by-instance sense affects their decision-making.
Critical strengths
Training a per-person model is an important problem that is relevant to many decision-making contexts. In settings such as chess where there is rich temporal data, using the timing information is an important potential signal.
Critical weaknesses
Despite a subsection in the Introduction entitled "Contributions" with a numbered list, the paper has difficulties making what its nove contributions actually are relative to the existing body of literature that builds individual-level models and incorporates timing information. The overview of the contributions in the subsection in the Introduction entitled "Contributions" begins with a summary sentence that doesn't convey what the paper actually accomplishes: "The finding is a tension: measurable personal differences persist even where their added value per decision is small, and against the game’s own clock history a static person term adds only a small accuracy signal."
Very little of the main body of the paper provides insight into the comparison with the large body of related work introduced in Section 2. If Table 1 at the bottom of page 7 is supposed to be an important place for seeing the benefits of the approach in the current submission, it needs to be expanded upon and explained much more clearly.
All of these attempts at explaining the contributions are hampered by an extremely compressed writing style that doesn't surface the key new points that the current paper brings to the problem.
Questions for authors
Since the paper does not succeed in providing a succinct description of its contributions, could you answer the following two questions: (1) What is the specific method by which you are evaluating your approach relative to the approaches in past work? (2) What is the concrete evidence via this method that your approach represents an improvement over existing approaches to the problem?
what is new, the evidence, and the comparison with prior work
The paper does not claim a better person model. It combines standard tests with two controls and a known-trait ceiling to assess what a person term adds, and shows cases where they change a reading.
Your first question: the method for evaluating against past work. For chess timing and solving, every comparator is scored on the same held-out decisions, against the same frozen population model, with paired player-level intervals. Earlier approaches enter as rows of Table 1: population models (Maia-2 through our reference; Allie on its blitz subset, calibrated on this cohort), simple per-person models (a tuned tempo offset; a joint speed–ability model, post hoc) and models that read recent history (this game's clock record; the person's recent solves). Table 55 (Appendix Y.11) names the analysis that stands for each line of prior work. For moves we use our own adaptations of Maia-2 and Maia-3 (Table 2); published per-player models enter only through their reported numbers.
Your second question: concrete evidence of improvement. Over the population model: yes, but small. The person vector improves think-time prediction by 0.0134 nats per decision over our Maia-2-based reference, below our 0.02 margin (Table 1). Over existing person models: no. A tuned one-number tempo offset is not detectably worse (difference 0.0030 nats [−0.0012, 0.0072]; exploratory). What the paper adds is the second test's two controls, and three cases show why they matter.
(i) A published comparison. From about twenty games per player, Maia4All reports top-1 accuracy of 53.22% against 51.46% for Maia-2, and 51.89% for a strength-only variant (110 players, ten above 2000; table on page 3). The strength-only variant is itself fitted on each player's games. That comparison has no arm fitted without the player's data, so its gain cannot be split into a part that needs the player's data and a part that does not. Our second test asks for that arm.
(ii) Our move data (Table 2; exploratory). For new blitz players rated 2000 or above, a correction to Maia-2 with no per-player parameters, fitted on other players' games, gives 0.74 of what our per-player embedding gains. For players rated 1100 to 1499 it gives 0.11. The answer also depends on the population model: one correction recipe gains 0.0176 nats per move on Maia-2 and 0.0015 on the newer Maia-3 (post hoc; Maia-3's training period covers these months). Our adaptations use about a hundred games per player and are no test of published per-player models.
(iii) The tests can say yes (Table 3; exploratory). In three of five further data sets, a person term clears the held-out margins against a correction fitted without the person; in a fourth, a tutoring log, it clears them over the reference (Table 3, note c). It does not against the person's recent record, where one exists.
Novelty against prior individual-level models. Each test alone is standard, and Section 2 credits its sources. New is their joint use: two controls and a ceiling, the gain a perfectly known trait would give. The ceiling yields a finding of its own. In chess the gains reach about two-thirds (tempo) and 76% (ability) of what a perfectly known static trait would give, and a known ability would still fall short of its margin (Section 5.2). The controls also changed our own readings: in Table 4, four of our plain comparisons shrank or vanished under a control (exploratory).
The summary sentence of the contributions. We agree, and have removed that sentence. The Contributions paragraph now opens: "We give tests that detect when a gain over a population model overstates the value of a person term, and apply them to six data sets."
Comparison with related work in the main body. A table on page 3 now maps each line of prior work to the analysis here that tests it, with its result. Sections 5.2 and 5.5 then make the comparisons: per-person offsets and history-aware models for timing; Maia-2, Maia-3 and Maia4All's published comparison for moves.
Table 1. Table 1 now reads positive as better. Its first number column is each comparator's timing gain over the reference. The next two give what the person vector adds beyond that comparator, for think time and for solving. For rows marked †, it is what the vector adds when refitted on top of the comparator. Section 5.2 walks through the rows.
Compressed writing. We agree. The abstract now opens with the question, each results subsection (5.1 to 5.5) opens with its answer, and most numbers of Section 5.5 sit in Tables 2 to 4.
Thank you for the detailed, thoughtful review. We reply late because we revised the paper thoroughly rather than answer in haste. Every point you raised is taken up, and the paper is revised throughout. We would be grateful if you read the revised version.