acceptodds
Under review as a conference paper at ICLR 2027

Closing the Loop on Human Behavior Prediction

Abstract

LLM digital twins are usually scored against how closely people reproduce their own answers on a retest, as if retest agreement were a ceiling on individual accuracy. It is a floor: the accuracy of copying the earlier answer, hence a lower bound on the best accuracy attainable from information that includes that answer, which the nearest-neighbor inequality of Cover and Hart bounds from above under stable propensities. We score LLM and non-LLM predictors of individual decisions on identical rows in the closed loop, where a predictor sees the person's earlier behavior and observes each decision after predicting it, on the Twin-2K-500 retest wave, three waves of the General Social Survey, Psych-101 and held-out Psych-201 experiments. Most of the individual signal our predictors extract lies in the person's most recent relevant behavior, and they differ in whether they anticipate a departure from it. On Twin-2K-500, dividing by retest agreement credits the item mode with more than the dataset's default twin under the metric of the dataset's authors, every released twin falls below the item mode, and an LLM given the earlier answer matches copying only by repeating it. On cognitive tasks, conditioning on the previous choice is the largest gain for statistical predictors, which then predict repeats, and a recurrent network trained on other persons' choices and outcomes reaches the accuracy of the best LLM on Psych-101. Reading the person's raw transcript, gemini-3.6-flash detects more switches than CAOP-sel and the transition predictors at the same false-alarm rate, but on held-out Psych-201 experiments it is right on switches only slightly more often than the item mode. Predicting when and how a person departs from recent behavior remains open.

Primary area:
applications to neuroscience & cognitive science
Keywords:
human behavior prediction, digital twins, large language models, test-retest reliability, Bayes error, nearest-neighbor inequality, online sequential prediction, cognitive modeling, panel surveys, benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_T9TmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper looks at how we evaluate models that predict what an individual person will do, with LLM "digital twins" as the main target.

The tasks are concrete. On Twin-2K-500, people answered a long survey and were asked some of the same questions again about two weeks later; the model has to predict the second answer from everything the person said before. The paper does the same with three waves of the General Social Survey, and with sequences of choices in cognitive experiments (Psych-101/201). Statistical models, a small RNN and prompted LLMs are all scored by per-person accuracy, and each is shown next to two simple baselines: the most common answer among other people, and simply repeating the person's earlier answer.

The central argument is that the common practice of reporting twin accuracy as a fraction of test–retest agreement is misleading. Retest agreement is exactly what you get by copying the earlier answer, so for a model that sees that answer it is a floor rather than a ceiling. The authors derive an upper bound from classical results. Empirically, all 13 released twins score below the most-common-answer baseline, LLMs that are given the earlier answer mostly just copy it, and on the cognitive tasks the previous choice carries most of the predictive signal.

Critical strengths

The main point is simple and, I think, important. Under the usual normalisation, even "guess the most common answer" would look like 92.6% of the ceiling, and every released twin actually falls below that baseline. That is a striking result, and it should change how digital-twin papers report their numbers. It is the main reason I lean towards acceptance.

The work is careful. As far as I could check, the theory is correct and the authors are upfront about where it stops holding. I also liked the controls: all models are scored on the same rows, the earlier answer is deleted from the LLM prompt to see what remains, and shuffled versions are used as sanity checks.

Critical weaknesses

Novelty. The bounds themselves are classical (Cover–Hart, classical test theory), and the paper says so, so the contribution is mostly in applying them to digital twins. The related work covers inter-annotator and neuroscience noise ceilings, but I was surprised not to see the "magic barrier" from recommender systems (Herlocker et al., 2004; Said et al., 2012), where users' inconsistency when re-rating items was used in much the same way to bound achievable accuracy. A short comparison would help, as would a clearer statement of what is new beyond Props. 3 and 6 and the empirical audit.

The framing is stronger than the evidence. The abstract leads with "retest agreement is a floor", but no model in the paper beats copying by more than about a point (0.4 on Twin-2K-500, 1.2 on GSS). The released twins being criticised never see the earlier answer, so for them the fair comparison is the most-common-answer baseline, not copying. And the line-deletion test showing that the LLM is copying uses one model and 150 people. The limitations section concedes most of this; I'd like the abstract to as well.

Questions for authors

How does your bracket relate to the magic barrier in recommender systems, and what would you say is new beyond Props. 3 and 6 and the audit?

Could the abstract lead with the difference between models that see the earlier answer and models that don't?

Do the main results hold without the repeated pricing questions? They make up about half of the Twin-2K-500 rows, and people may simply remember what they said last time.

Does the full-persona copying result replicate with a second LLM and with a prompt that invites the model to consider a changed answer?

4 comments
AuthorsOpenReview ↗

Appreciation for the comment and some further discovery shared

Dear Reviewer,

Thank you for finding the normalization point important and for checking the theory and the controls. That point follows from the paper's method, which makes explicit what each predictor sees. In this method, a predictor sees a person's earlier behavior and observes each decision after predicting it, a setting that earlier work calls prequential.

Your first question asked how our bounds on the best attainable accuracy, the retest bracket, relate to the magic barrier. The barrier is the squared-error counterpart of what the bracket bounds from above, and not of retest agreement. The bracket starts at the accuracy of copying a person's earlier answer, which is retest agreement. It ends at a bound on the accuracy of a predictor that knows each person's answer probabilities, if these are stable. The paper calls this predictor the oracle and the bound the upper end. The magic barrier is likewise the error of a predictor that knows each user's expected rating. If two ratings are independent draws from the same distribution, copying one to predict the other has exactly twice the squared error of that predictor. One re-rating thus identifies the barrier under squared error. Estimates that report the error between two ratings as the barrier, as in Amatriain et al. 2009, therefore report the error of copying, the counterpart of dividing by retest agreement. Under accuracy, retest agreement likewise measures copying, but no exact relation links copying to the oracle, so one retest only brackets the oracle. The related work on page 4 now makes this comparison.

Appendix A.2 estimates the barrier on the 43 items of Twin-2K-500 with ordered options, coded from 0 to 1. With stable answer probabilities, its square is estimated at 0.028, half the squared error of copying, 0.056. Predicting instead the average retest answer of other persons with the same earlier answer lowers the squared error to 0.043.

You also asked what is new beyond Propositions 3 and 6 and the results on twins. What is new is the method, which extends this prequential setting to survey retests and matches each reference to what the predictor sees. For a predictor that sees the earlier answer, retest agreement is a floor. A twin does not see the earlier answer, so the simplest reference matched to it guesses the most common answer to the question, which the paper calls the item mode. The revised Discussion therefore asks evaluations of twins to report the item mode on the same rows. Beyond twins, variants of each statistical predictor separate the previous choice from learning on cognitive tasks.

We agree that the upper end itself comes from the classical inequality of Cover and Hart and its weaker form from test theory. Beyond them, Proposition 5 keeps the upper end when answer probabilities drift in either of two ways the predictor cannot foresee. Remark 2 keeps it under a memory that repeats any earlier answer at the same rate. Under the same conditions, retest agreement is a ceiling only for a predictor that guesses at random with the true answer probabilities given what it sees, Remarks 1 and 4. If every person answers yes with probability 0.8, such a predictor has accuracy 0.68, the retest agreement, but always predicting yes has accuracy 0.8.

Your second question asked whether the abstract could lead with the difference between models that do and do not see the earlier answer, and it now does. It opens with predictors that see the person's earlier behavior, for which retest agreement is a floor. It then turns to twins, which do not see the earlier answer, and to the item mode, which all released twins fall below. The abstract also states the limit you named, that no predictor we ran beats copying by more than 1.2 points on either survey.

Your third question asked whether the main results hold without the repeated pricing questions, and they do. On the 68 other items of Twin-2K-500, copying beats the item mode by 10.5 points, as on all rows. All 13 released twins stay below the item mode. Under the metric of the dataset's authors, dividing by retest agreement credits the item mode with 93.0% instead of 92.6%, Section 6.1 and Appendix A.2.

Your fourth question asked whether the full-persona copying result replicates with a second LLM and with a prompt inviting a change, and it does. On 10 retest rows of each of 450 persons, GPT-6.1 Sol, given all the person's earlier categorical answers, repeats the earlier answer on every row. The first LLM, deepseek-flash, repeats it on 98.6% of them. A second prompt adds that about 29% of people change their answer and asks whether this person would. Under it, GPT-6.1 Sol still repeats on 99.98% of the 4,500 rows and deepseek-flash on 96.7%. Deleting the prompt line with the earlier answer costs GPT-6.1 Sol 7.4 points and deepseek-flash 9.3, Section 6.3 and Appendix F.

We would welcome your view on whether these answers resolve the two weaknesses you named.

Dear Reviewer,

Thank you again for your review and for the pointer to the magic barrier. We would be grateful for your view on whether our response and the revision answer your four questions. On the first, the magic barrier is the error of a predictor that knows each user's expected rating. It thus corresponds to the accuracy of a predictor that knows each person's answer probabilities, which our upper bound limits, and not to retest agreement. On the second, the abstract now leads with the difference between predictors that do and do not see the earlier answer. It also states that no predictor we ran beats copying that answer by more than 1.2 points on either survey. On the third, the main results hold without the repeated pricing questions. On the fourth, the full-persona copying result replicates with a second LLM and with a prompt inviting a change. We are glad to answer any further question.

Thank you for the response in addressing my questions. I'd like to maintain my rating.

Dear Reviewer,

Thank you for the reply and for your review. We are pleased that we address your questions. Your suggestions led to the comparison with the magic barrier, the analysis without the repeated pricing questions and the replication with a second LLM. The paper is stronger for them.

Reviewer_YWgbOpenReview ↗
Rating
1
Confidence
3(+2)
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a closed-loop protocol for predicting individual decisions: a predictor sees prior behavior, predicts the next choice, and then observes the true outcome. Using survey retest data, it shows retest agreement lower-bounds attainable accuracy and finds that most individual signal lies in a person’s most recent relevant behavior.

Critical strengths
  1. Public data and released code; all predictor families are scored on identical rows.
  2. The ablation showing that the LLM reaches copying accuracy only by repeating the earlier answer (0.716 → 0.617 when that line is removed) is a useful observation about how persona-based twins individuate.
Critical weaknesses
  1. The purpose of the retest in the closed loop is unclear. In survey research, a retest measures test-retest reliability, i.e., how consistently a person answers the same question. Digital-twin studies use it the same way (Park et al., 2026; Toubia et al., 2025): the twin predicts the retest answer without seeing the earlier answer, and the person's own retest agreement serves as a reference for the twin's accuracy. In this paper, the predictor receives the earlier answer Y(1)Y^{(1)} as input and predicts Y(2)Y^{(2)}. The retest then neither measures the person's consistency nor serves as a reference; it becomes the prediction task itself. The paper does not describe a setting where this task is useful: once Y(1)Y^{(1)} is available, the question has already been answered. The results reflect this. The survey benchmark thus measures retest agreement a second time. It reads as an audit of current twin evaluations, not as a benchmark of predictors.
  2. Assumption 1 does not hold for human responses, and the main theoretical contribution depends on it. Assumption 1 treats the two answers as independent draws from one fixed propensity. Human responses depend on context that changes over time and across situations, so the information a predictor is missing is not fixed but varies from one occasion to the next (Peng et al. 2026). Without Assumption 1, neither the upper end nor the reading of retest agreement as a ceiling for replicas is identified. What remains is the lower end, which only says that copying the earlier answer is a feasible predictor.
Questions for authors
  1. Please give a concrete deployment scenario in which predicting Y(2)Y^{(2)} given Y(1)Y^{(1)} on the same item has practical value. Without one, would you reframe the survey part as an evaluation audit rather than a benchmark?
  2. Your own tests reject stable propensities for a large share of items. What does the retest bracket tell us when the person's state differs between the two occasions?
5 comments
AuthorsOpenReview ↗

Response on Q1 and further illustration on the topic

Dear Reviewer,

Thank you for noting the public data and code, the scoring of all predictors on identical rows and the ablation that deletes the earlier answer. This first comment answers Q1 and then sets out the paper's argument step by step. A second comment answers Q2.

On Q1, one deployment is imputation in panel surveys, which fills a missing answer from the person's earlier answers. Some panel surveys simply copy the earlier answer, known as carrying it forward, Watson and Starick 2011. Yet the earlier answer does not settle the later one. On the Twin-2K-500 survey, people change about 29% of their answers within weeks, and copying gets every changed answer wrong. On the General Social Survey, copying reaches an accuracy of 0.659 on the answers given two years later. Tutoring systems make a prediction of the same kind, whether a learner will recall an item practiced before, Settles and Meeder 2016. The revised introduction names these settings on page 2.

On reframing, the revised paper puts its evaluation method to two uses on surveys, each with its own reference, page 10. The first use concerns predictors without the earlier answer, such as LLM digital twins, and compares them with guessing each question's most common answer. This use covers the scoring of twins that your review reads as an audit. It also covers other predictors, and on Twin-2K-500 a statistical model of the person's other answers is 2.7 points above that guess. The second use concerns predictors with the earlier answer. Their reference is copying, whose accuracy is retest agreement, the rate at which people repeat their own earlier answers. This use shows whether a predictor does more than repeat the earlier answer, as in the ablation you note, and whether it anticipates who changes an answer. No predictor we ran beats copying by more than 1.2 points on either survey, so anticipating who changes remains open.

Whether a predictor can beat copying depends on why answers change, and two classical readings of unstable answers disagree. On the first, people err around a most likely answer at a common rate, and a predictor that knows this answer beats copying, Achen 1975. On the second, people either always repeat or answer at random, and no predictor beats copying, Converse 1964. On 393 of the 465 binary items of Twin-2K-500, one retest cannot tell the two readings apart, page 6.

The argument behind these answers starts from the method. The paper proposes a method for evaluating predictions of a person's next answer or choice. In this method, a predictor sees the person's earlier behavior and observes each decision after predicting it, a setting the paper calls the closed loop. The paper applies it to surveys with a retest and to cognitive tasks.

In a survey with a retest, a predictor that sees the earlier answer can copy it. Copying is right exactly when the person repeats the answer, so its accuracy is retest agreement. For such a predictor, retest agreement is therefore a floor on the best attainable accuracy, with no condition needed.

Above this floor, Section 4 bounds the best attainable accuracy from above, by what the paper calls the upper end. The upper end needs a condition on how a person's two answers relate, which our second comment discusses. The interval from the floor to the upper end is the retest bracket, and on Twin-2K-500 it runs from 0.712 to 0.828 under that condition.

Making explicit what the predictor sees also shows why reading retest agreement as a ceiling for an LLM digital twin misleads. A twin does not see the earlier answer and so cannot copy it. A fair reference has no more information about the person than the twin, so it must not see the earlier answer either. The simplest such reference guesses the most common answer to the question, which the paper calls the item mode. On Twin-2K-500, dividing by retest agreement credits even the item mode with 92.6% under the metric of the dataset's authors. All 13 twins released with the dataset fall below it.

Applied to two surveys and two collections of cognitive experiments, the closed loop shows that most of the individual signal our predictors extract lies in the person's most recent relevant behavior. On the surveys, this behavior is the earlier answer. On Twin-2K-500, the ablation you note shows that an LLM matching copying does so only by repeating the earlier answer. The revision extends this ablation to a second LLM from another provider. GPT-6.1 Sol repeats the earlier answer on all 10 scored rows of each of 450 persons and loses 7.4 points when the prompt line holding it is deleted. Without that line it stays 4.0 points above the item mode.

On cognitive tasks, which have no retest, the previous choice plays the role of the earlier answer. Statistical predictors gain much more from conditioning on it than from updating their parameters after each choice.

AuthorsOpenReview ↗

[Continued] Response on Q2

This second comment answers Q2, what the retest bracket tells us when a person's state differs between the two occasions. In short, the bracket still bounds attainable accuracy from below, and still bounds it from above for two kinds of change that the predictor cannot foresee. The bracket is an interval for the best attainable accuracy of a predictor that sees the person's earlier answer. Its floor is the accuracy of copying that answer, which is exactly retest agreement, the rate at which people repeat their own earlier answers. This floor needs no condition, since any predictor that sees the earlier answer can copy it. Nor do the conclusions built on the floor and on measured accuracies. These include the comparison of LLM digital twins with guessing each question's most common answer, the ablation that deletes the earlier answer and all results on cognitive tasks.

The floor also says more than that copying is feasible. It identifies retest agreement as the accuracy of copying the earlier answer, which twins do not see. Dividing a twin's accuracy by retest agreement therefore compares predictors with unequal information.

The bracket ends at a bound that the paper calls the upper end. Unlike the floor, the upper end needs a condition, but it can survive a change of state. In a worked example, every person answers yes to an item with probability 0.8, independently each time. Copying is right with probability 0.82+0.22=0.680.8^2+0.2^2=0.68, the retest agreement. A predictor that knows the probability always says yes and is right with probability 0.8, exactly the upper end of Section 4 computed from 0.68. Now let the probability move on the retest to 0.9 for half of the persons and to 0.7 for the other half, in a way the predictor cannot foresee. This change is one instance of what your review describes, a context that varies between occasions and that the predictor is missing. Under this change, retest agreement becomes 0.74 or 0.62, still 0.68 on average. The best predictor still says yes and is right with probability 0.8 on average, so the same upper end holds, Appendix C.6.

Proposition 5, already in the appendix of the submission, proves this for two kinds of unforeseen change. One kind averages out among persons who start with the same answer probabilities, as in the example. The other moves everyone at a common rate toward the population answer shares, the fraction of all persons giving each answer. For both kinds, each answer must still be an independent draw given the answer probabilities. Under these conditions, the new Remark 4 also keeps retest agreement as a ceiling for a predictor that guesses at random with the true answer probabilities given what it sees. The paper calls such a predictor a replica.

A change can also be foreseen, since in the paper's method the predictor observes each retest answer after predicting it. Answers already revealed in the retest wave may thus expose a change in the person's answer probabilities, one form of the changing context you raise, citing Peng et al. 2026. The new Remark 5 gives a separate bound for a predictor that foresees the change in this way. On the Twin-2K-500 survey the revealed answers help little, since updating our most accurate predictor on them changes its accuracy by only -0.01 points, 95% interval [-0.07, 0.04].

Our tests do reject equal answer shares for many items, as you note. On Twin-2K-500, the answer shares of the two occasions differ at the 5% level for 47% of the 43 items with more than two options. They differ for 6% of the 465 binary items, near the 5% expected by chance. If every person's answer probabilities shifted by the observed change on the items where the test rejects, the upper end would rise from 0.828 to 0.836, page 6.

The General Social Survey interviewed three panels of persons in three waves each, two years apart. A test that the three waves are interchangeable rejects at the 5% level for 44% of the items, page 8. The data show the pattern that a gradual change predicts. The agreement between the two waves four years apart is the lowest of the three agreements for 70% of the items, against one third under stable answer probabilities. A third wave separates a change toward the population answer shares from unreliable answers, as Heise 1969 did for correlations, Proposition 6. Suppose the items of each panel share one rate of change. If the predictor cannot foresee the change, the upper end for persons who answered all three waves falls from 0.797 under stable answer probabilities to an estimated 0.785. If it foresees the change to the third wave and waves 2 and 3 are equally reliable, Remark 5 puts the upper end at an estimated 0.809, Appendix C.4.

We agree that without any condition relating the two occasions no upper bound below 1 exists, Remark 3. Retest agreement read as a ceiling is such a bound, so it needs a condition as well.

We would welcome your view on whether these answers resolve Q1 and Q2.

Dear Reviewer,

Thank you again for your review. We would be grateful for your view on whether our two comments and the revision address Q1 and Q2. On Q1, some panel surveys fill a missing answer by copying the earlier one. Yet people change about 29% of their answers within weeks on Twin-2K-500, and copying gets every changed answer wrong. Predicting who changes therefore has practical value. On the reframing in Q1, the method has two uses on surveys. It scores predictors without the earlier answer, such as twins, against guessing the most common answer, and predictors with the earlier answer against copying it. On Q2, the lower bound on attainable accuracy needs no condition, since a predictor that sees the earlier answer can always copy it. The upper bound does need one, and Proposition 5 keeps this bound when a person's state changes in either of two ways the predictor cannot foresee. We are glad to answer any further question.

Reviewer_4sC4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

This paper explores the closed-loop behavior prediction problem. It demonstrates the retest bracket with a theoretical analysis, where copying the previous answer is just a lower bound of this bracket. With datasets on social surveys and cognitive experiment records, this paper further benchmarks LLMs together with statistical, RNN, and other methods. Experiment results suggest that most of the signal can be extracted in the individual's recent relevant behavior and LLMs tend to repeat the earlier answers.

Critical strengths
  1. The benchmark contains multiple datasets, including both social surveys and cognative experiment records. The data are also split into folds and some been retrained as held-out. The construction of evaluation data is rigourous.
  2. Multiple types of behavior predictors are tested, including not just LLMs, but also statistical predictors, RNN, and CAOP. This design provides information on comparing how LLM predicts behavior versus other methods. Different prompting strategies are also been tested for LLMs.
  3. Section 4 provides theoretical analysis on the upper end of the retest bracket, demonstrating the potential room for improving the prediction.
  4. The finding from the experiment results is interesting: most of the signal can be extracted in the individual's recent relevant behavior and LLMs tend to repeat the earlier answers.
Critical weaknesses
  1. The writing is difficult to read and follow. The presentation is full of long sentences, and many of them spread across 5 or even 6 lines. Information and new concepts are also often introduced without supporting contexts (e.g., "batch/state/online" in Fig 1a and referred by the introduction).
  2. The stable propensity asssumption (Assumption 1) might be too strong, making the derived retest bracket proposition questionable. First, it assumes each person has a stable latent state θp\theta_p, but one person's status might change overtime, so assumping a changing state θpt\theta_{pt} seems to be more appropriate. Also, it assumes that all answers of person pp are independent, but this might be untrue for many cognitive experiments and social survey items. The authors need to further justify the Markovian property of the behavior prediction task to make such an assumption, or to prove that proposition 1 still holds without assumption 1 (or a loose version of it).
Questions for authors

Please refer to the Critical Weaknesses section.

4 comments
AuthorsOpenReview ↗

Response to the concerns and some further illustration

Dear Reviewer,

Thank you for noting Section 4's analysis, the finding about recent behavior, the rigorous evaluation data and the range of predictors. The analysis and the finding come from the paper's method. In this method, a predictor sees a person's earlier behavior and observes each decision after predicting it, a setting the paper calls the closed loop.

We answer your two concerns in turn, readability and the strength of Assumption 1. On readability, we agree and have split the long sentences of the main text. Pages 1 and 2 now define the main terms, including your example, the batch, state and online variants of Figure 1a. The batch variant of a statistical predictor ignores the revealed choices. The state variant conditions on the most recent one, and the online variant also updates its parameters after each choice.

On Assumption 1, you asked us to justify the Markov property or to prove Proposition 1 under a looser assumption. Proposition 5 does the second, and we report what the data say about the first. Proposition 1 gives a lower and an upper bound. It concerns a person who answers the same item twice and a predictor of the retest answer. Copying the earlier answer is right exactly when the person repeats it, so its accuracy is the share of repeated answers, called retest agreement. A predictor that sees the earlier answer can copy it, so for such a predictor retest agreement is a floor. This floor is the lower bound, and it needs no assumption. Most of the paper's results rest on it and on measured accuracies. They include the finding about recent behavior that you noted and the test that deletes the earlier answer from an LLM's prompt.

The upper bound says how far a predictor could beat copying, and the paper calls it the upper end. Unlike the floor, the upper end needs Assumption 1 or a looser assumption. Any upper bound below 1 needs some assumption, Remark 3. Section 4 now states on page 6 which claim needs which assumption.

Your first point is that a person's state may change over time, which you write as θpt\theta_{pt}. A change of state alters the person's answer probabilities between the two occasions. Proposition 5, already in the appendix of the submission, proves that the upper end still holds for two kinds of such change that the predictor cannot foresee. One kind averages out among persons who start with the same answer probabilities. The other moves everyone at a common rate toward the fraction of all persons giving each answer. To see the first kind, let every person answer yes with probability 0.8, independently each time. Copying is then right with probability 0.82+0.22=0.680.8^2+0.2^2=0.68, which gives an upper end of 0.8. If the probability moves on the retest to 0.9 for half of the persons and to 0.7 for the others, copying is still right with probability 0.68 on average. The best predictor says yes for everyone and is right with probability 0.8 on average, so the upper end still holds, Appendix C.6.

With a third occasion, the rate of the second kind of change can also be estimated, Proposition 6. The General Social Survey interviewed three panels of persons three times each, two years apart. With one rate per panel, the estimated upper end for persons who answered all three times moves from 0.797 under Assumption 1 to 0.785, page 8.

Your second point is that a person's answers may depend on each other. In cognitive experiments, choices can indeed depend on earlier feedback, so the paper uses no bound there, Section 6.2. On surveys, Assumption 1 makes answers independent only given the person's state. Through that state they are dependent, which is why the earlier answer predicts the retest answer, and page 5 now says so. The proof needs even less, only two things, Appendix C.2. Given the person's state, both answers to the predicted item have the same distribution, and the retest answer is independent of what the predictor sees. Neither restricts how the other answers depend on each other. Earlier answers may therefore depend on each other in any way. The independence does exclude a memory of the earlier answer. Yet the upper end survives a memory that repeats the earlier answer at a rate that does not depend on that answer, Remark 2.

We read the Markov property you asked about as this independence. Given the state, what the predictor sees says nothing more about the retest answer. The Twin-2K-500 survey allows a check. A person who changed more of the retest answers revealed so far is more likely to change the next one. This can reflect that some persons are less consistent than others, which the assumption allows, as well as a dependence that it rules out. Either way, updating our most accurate predictor on the revealed answers changes its accuracy by only -0.01 points, 95% interval [-0.07, 0.04], page 6.

We would welcome your view on whether pages 1 to 6 address both concerns.

Dear Reviewer,

Thank you again for your review. We would be grateful for your view on whether the revision and our response address your concerns. On Assumption 1, the lower bound on attainable accuracy needs no assumption, since a predictor that sees the earlier answer can always copy it. The upper bound does need one, and Proposition 5 keeps this bound when a person's state changes in either of two ways the predictor cannot foresee. Under Assumption 1, a person's answers are independent only given the person's state. Through that state they depend on each other, which is why the earlier answer predicts the retest answer. We are glad to answer any further question.

I appreciate the authors' efforts in revising the manuscript substantially and addressing my concern on the assumption. I have raise up my score accordingly.

Dear Reviewer,

Thank you for reading the revision and the response. We are pleased that the treatment of the assumption of stable answer probabilities addresses your concern.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the revision

Dear Area Chair and Reviewers,

We thank the reviewers for all the comments. The paper proposes a method for evaluating predictions of a person's next answer or choice. The predictor sees the person's earlier behavior and observes each decision after predicting it, a setting the paper calls the closed loop. This setting is already used in knowledge tracing. The paper applies it to retest data, where people answer the same question again. On such data, the paper bounds the best attainable accuracy from below and above, and calls the interval the retest bracket. The bracket starts at the accuracy of copying the earlier answer, which is exactly retest agreement, the rate at which people repeat their own earlier answers. A predictor that sees the earlier answer can copy it, so for such a predictor retest agreement is a floor, with no condition needed. The bracket ends at a bound given in Section 4, which the paper calls the upper end. This upper end needs a condition on how a person's two answers relate. The simplest such condition, Assumption 1, is that both answers are drawn independently from the same answer probabilities.

The closed loop also covers cognitive tasks, which have no retest, and there the previous choice plays the role of the earlier answer. In this loop, each statistical predictor comes in three variants. One ignores the revealed choices, one conditions on the most recent of them, and one also learns by updating its parameters after each. The paper calls them the batch, state and online variants, and they separate the previous choice from learning. Conditioning on that choice is the largest gain for statistical predictors, and learning adds little. The loop also separates predictors on the choices where a person switches away from the previous choice. At an equal rate of false alarms on repeated choices, an LLM that reads the person's transcript, gemini-3.6-flash, detects 78% of these switches in the Psych-101 experiments. A lightweight online baseline detects 69%, Table 1.

Making explicit what the predictor sees also shows that a common way of scoring LLMs that simulate a person, called digital twins, is misleading. Twins are often scored against retest agreement as if it were a ceiling on accuracy. A twin, however, does not see the earlier answer and so cannot copy it. A fair reference has no more information about the person than the twin, so it must not see the earlier answer either. The simplest such reference guesses the most common answer to the question, which the paper calls the item mode. On Twin-2K-500, dividing by retest agreement credits even the item mode with 92.6% under the metric of the dataset's authors. Dividing the same way credits their default twin with 88.4%, which looks strong, yet this twin scores below the item mode. All 13 twins released with that dataset score significantly below the item mode. This comparison needs no condition, since both accuracies are measured.

Section 4 now states which claim needs which condition. Beyond Assumption 1, the upper end also holds when a person's answer probabilities drift in one of two ways the predictor cannot foresee. The drift averages out among persons who start with the same answer probabilities, or it moves everyone toward the population answer shares at a common rate. Proposition 5 proves this, and it was already in the appendix of the submission. The new Remark 5 also bounds a predictor that can foresee the drift. The related work now compares these bounds with a limit that inconsistent ratings impose on recommender systems, known as the magic barrier.

Beyond the theory, new analyses support the empirical results. One concerns LLMs whose prompt holds the person's earlier answer to the same question. In the submission, the closed loop showed that one such LLM matched the accuracy of copying only by repeating that answer. On 10 retest rows of each of 450 persons of Twin-2K-500, a second LLM from another provider, GPT-6.1 Sol, repeats the earlier answer on every row. It loses 7.4 points when the prompt line holding that answer is deleted, Section 6.3.

Another analysis drops the repeated pricing questions, about half of the retest rows of Twin-2K-500, because people may remember their earlier answers to them and so inflate copying. Without them, copying still beats the item mode by 10.5 points, and all 13 released twins stay below the item mode.

The closed loop also makes it measurable whether a predictor anticipates which persons will change an answer. No predictor we ran does this well, so the problem remains open. Such predictions matter in settings that the introduction now names. Some panel surveys, for example, fill a missing answer by copying the earlier one and so get every changed answer wrong.

These additions are printed in blue in the revised PDF. None of them changes a result of the submission. In the main text, long sentences are also split, and pages 1 and 2 define the main terms.

AuthorsOpenReview ↗

Closing Summary

Dear Area Chair and Reviewers,

We thank the reviewers for the discussion. Predicting what an individual person will answer or choose from that person's earlier behavior is the task of knowledge tracing and of LLM digital twins. For such a predictor, the accuracy it can attain and the reference it should be compared with both depend on what it sees about the person. The paper develops a method from this observation. The method matches each predictor with a reference that sees no more about the person. It bounds the accuracy the predictor can attain, from below without any assumption and from above under a stated one. It also runs each statistical predictor in variants that separate what the predictor gains from the person's most recent behavior from what it gains by learning.

The matched reference and the lower bound need no assumption. A predictor that sees the person's earlier answer to the same question can copy it, and the accuracy of copying is exactly retest agreement, the rate at which people repeat their own earlier answers. For such a predictor, retest agreement is therefore a lower bound on the attainable accuracy and not a ceiling. A predictor that does not see that answer, such as the twins released with Twin-2K-500, cannot copy it, so its reference must not see the answer either. The simplest such reference guesses the most common answer to the question. Twins are nevertheless often measured against retest agreement, and under the metric of the dataset's authors this credits even the guess with 92.6%. All 13 released twins score below the guess, also without the repeated pricing questions.

The upper bound does need an assumption. When a person's two answers are independent draws from the same answer probabilities, which is Assumption 1, copying is the nearest-neighbor rule, and the inequality of Cover and Hart bounds what any predictor with this information can attain. Proposition 5 keeps this bound for two kinds of change that the predictor cannot foresee. For other changes the bound can be higher, by an amount that two answers per person do not identify. Under Assumption 1 the interval on Twin-2K-500 runs from 0.712 for copying to 0.828, and one retest cannot tell where in it the best attainable accuracy lies.

On cognitive experiments, the previous choice plays the role of the earlier answer, and the variants show that statistical predictors gain more from conditioning on it than from online learning. In Psych-101, people switch away from their previous choice in 54% of the scored decisions, so repeating it is right in fewer than half. An LLM that reads the person's transcript reaches an accuracy of 0.704 there, 5.4 points above the best statistical predictor, and a recurrent network trained on other persons matches it.

Starting from what a predictor sees, the paper gives each predictor a reference that sees no more about the person. For a predictor that sees the earlier answer, it adds a lower bound without assumption and an upper bound under a stated one. On Twin-2K-500, this shows that all 13 released twins, which do not see the earlier answer, fall below guessing the most common answer. LLMs from two providers that are given it match copying only by repeating it. On cognitive experiments, conditioning on the previous choice is the largest gain for statistical predictors, and an LLM that reads the transcript and a recurrent network beat them on Psych-101.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.