- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper combines labeled source-agent decisions, a few target-agent games, structured states, dialogue, and frozen LLM Observer representations in TabPFN to predict black-box agents’ next decisions without task-specific gradient updates. Bargaining and negotiation experiments show that correctly labeled target games improve prediction beyond additional source data.
Critical strengths
-
The study evaluates transfer from source LLM agents to independently scaffolded target agents with different prompts and control logic, providing a useful test of unfamiliar-agent prediction.
-
The wrong-target, shuffled-label, and extra-source controls support the claim that prediction benefits from correctly labeled behavior of the particular target. These experiments help distinguish target-specific adaptation from simply increasing the amount of conditioning data.
-
Comparisons with generic text encoders, Observer output probabilities, and controls for structured states and interaction trajectories help explain where hidden representations provide additional value.
Critical weaknesses
-
Although Section 4 explicitly constructs inputs from the deciding agent’s observable view, this includes the target’s own private payoff parameters, which an external counterpart may not observe. The paper should clarify the intended forecasting setting and either evaluate performance using only counterpart-observable inputs or qualify its practical claims accordingly.
-
The encoder and input-decomposition controls are useful, but generic encoders and the Observer still receive different inputs. Comparing them on the same serialized state and dialogue would better isolate the Observer representation’s value.
-
TabPFN receives up to 3,000 labeled source decisions unavailable to the direct LLM baseline, so their performance gap does not isolate predictor choice. A target-only TabPFN control would help.
Questions for authors
-
How do performance and Observer gains change when target-private parameters and their derived features are removed from both structured inputs and model prompts? Please report complete- and incomplete-information results separately.
-
Table 2 reports median R², but Appendix M defines error bars using σ/√N over target–seed pairs. Please provide paired confidence intervals for Observer gains that account for repeated evaluations of each target and, for proposals, recompute the median R² difference during resampling.
-
Please specify the information, history window, and pre-decision message cutoff available to each representation, and compare a generic encoder with the Observer on identical serialized inputs.
-
Please compare against target-only TabPFN and give the direct LLM baseline a manageable set of relevant source examples to assess the effect of unequal source supervision.
6 comments
Response to Reviewer bder (2/3)
Q2. "Please provide paired confidence intervals for Observer gains that account for repeated evaluations of each target and, for proposals, recompute the median R² difference during resampling."
Response. We now report paired cluster-bootstrap CIs for all Observer gains. In bargaining, every CI excludes 0, for responses and for proposals. In negotiation, response gains are significant at K≥2 and proposal gains are not.
Responses. We use a cluster bootstrap over targets (5,000 resamples), in the target view, with Gemma-2-2B layers 16–23. Team-clustered intervals are similar. AUC, game+text → +Observer:
| K | Bargaining | Gain [95% CI] | Negotiation | Gain [95% CI] |
|---|---|---|---|---|
| 0 | .640 → .668 | +.027 [.018, .037] | .812 → .820 | +.008 [−.002, .019] |
| 2 | .693 → .724 | +.031 [.022, .041] | .809 → .828 | +.019 [.005, .036] |
| 4 | .732 → .764 | +.032 [.023, .042] | .813 → .831 | +.018 [.006, .031] |
| 8 | .765 → .801 | +.036 [.026, .047] | .807 → .828 | +.021 [.007, .036] |
| 16 | .808 → .840 | +.033 [.024, .042] | .847 → .866 | +.019 [.005, .038] |
Proposals (Table 2, proposal panel). The submitted runs split at the decision level. We recomputed them with game-level splits, so no conditioning game appears in the test set in any of the 2,395 cells. We used:
- updated features, in the target view;
- PCA fit on conditioning rows only;
- the median-R² difference recomputed in each resample, as you requested.
All runs use Gemma-2-2B, layer 20. Median R², game+text → +Observer:
| K | Bargaining | Gain [95% CI] | Negotiation | Gain [95% CI] |
|---|---|---|---|---|
| 0 | .307 → .501 | +.194 [.087, .313] | .542 → .557 | +.015 [−.094, .117] |
| 2 | .410 → .586 | +.176 [.055, .264] | .600 → .654 | +.054 [−.039, .132] |
| 4 | .534 → .634 | +.100 [.043, .208] | .627 → .651 | +.024 [−.041, .094] |
| 8 | .612 → .695 | +.083 [.039, .152] | .728 → .760 | +.031 [−.043, .114] |
| 16 | .658 → .720 | +.061 [.032, .145] | .789 → .794 | +.005 [−.006, .065] |
The bargaining proposal gain holds at every K under the stricter protocol. The negotiation gain is not distinguishable from 0, as the paper states.
We also compared against the paper's direct LLM on identical test decisions, with both methods selecting context games using the same split function and seed. At K=16 in bargaining, TabPFN+Observer reaches a median R² of .800, against −.303 for the LLM. All CIs exclude 0, in both games and at every K.
We will add the paired confidence intervals for Observer gains to the paper and update the proposal panel of Table 2.
Response to Reviewer bder (3/3)
W2 / Q3. "Please specify the information, history window, and pre-decision message cutoff available to each representation, and compare a generic encoder with the Observer on identical serialized inputs."
Response. We added a representation from a stronger generic encoder (bge-m3) on exactly the same prompt. It adds nothing to the base configuration, and the Observer retains its gain. The table gives results after K=16 target games for bargaining / negotiation.
| Added to | None | + bge-m3 | + Observer |
|---|---|---|---|
| Game features + identity | .815 / .852 | .818 / .850 | .848 / .869 |
| Game features + text + identity | .807 / .849 | .816 / .848 | .840 / .864 |
Information available to each representation:
- Game features: configuration, the target's own parameter, the current offer, and the last five rounds.
- Dialogue embedding: current-round messages sent before the target's decision.
- Observer and bge-m3: the reconstructed decision-time prompt, covering the full game up to the decision.
We will add the comparison and the description of the information available to each representation to the paper.
W3 / Q4. "Please compare against target-only TabPFN and give the direct LLM baseline a manageable set of relevant source examples to assess the effect of unequal source supervision."
Response. A practical advantage of tabular models is that they can use many context rows at low cost. Providing the same amount of context to an LLM is expensive. We therefore tested the tabular model with the Observer when both methods receive the same small source sample. Our method leads when both systems receive one labeled decision from each of 16 source games and observe 16 target games.
We ran Gemini 3.5 Flash-Lite with K=0 and K=16 target games on the full target pools.
Matched source supervision. We gave both systems the same 16 labeled source decisions and the same K target games and evaluated them on the same test decisions. With no target games, the LLM is more accurate. After 16 target games, the tabular model with the Observer performs significantly better. The number of source labels alone therefore does not explain the advantage.
| AUC | Bargaining K=0 | Bargaining K=16 | Negotiation K=0 | Negotiation K=16 |
|---|---|---|---|---|
| Gemini 3.5 Flash-Lite | .692 | .735 | .657 | .732 |
| TabPFN, no Observer | .624 | .821 | .582 | .804 |
| TabPFN + Observer | .608 | .837 | .560 | .806 |
In a separate comparison using the submitted input view, we gave TabPFN 16 target games and no source decisions. It reached .799 AUC in bargaining and .803 in negotiation without the Observer, and .817 and .808 with the Gemma-2-2B layer-20 Observer. Adding the full source pool brought the scores with the Observer to .840 and .864.
Why does the LLM not receive all source rows? With 16 labeled source decisions, the prompts averaged 5,904 input tokens. TabPFN can use up to 3,000 labeled source decisions per prediction. Extrapolating from the measured number of tokens added by 16 examples, the 3,000 examples alone would take about 590,000 input tokens per prediction in bargaining and 635,000 in negotiation. At 0.40 USD per million input tokens, we estimate about 4,700 USD in input costs for the 19,294 test predictions in this comparison (K=0 and K=16).
We will add the comparison with matched source supervision to the paper.
We hope these analyses address your concerns. If any point remains open, we are happy to clarify or run further checks during the discussion.
Response to Reviewer bder (1/3)
Thank you for the focused review. We ran all four requested analyses:
Response. This is an important question. In bargaining, the Observer gain persists with public information alone. In incomplete-information negotiation, the estimated gain is smaller when the target’s own value is hidden. We will report both estimates in the abstract.
In §4.1 we will define the forecasting setting through two input views that differ in whether the forecaster knows the target’s own parameter:
Observer gain over game+text, by information regime: the range over K, and game+text → +Observer AUC at K=16.
Direct LLM under the public view. We reran Gemini 3.5 Flash-Lite without thinking on the same test decisions. In negotiation, hiding both valuations in incomplete-information games changed the LLM’s AUC from .502 to .464 at K=0 and from .734 to .718 at K=16. Public-view TabPFN with the Observer scored .723 and .835, respectively. In bargaining, the public-view and original LLM prompts were identical. The LLM scored .626 at K=0 and .737 at K=16, and public-view TabPFN with the Observer scored .691 and .854.
We will define both views in §4.1, add the public-view table to §6, and report the K=16 negotiation estimates for both views in the abstract: “In incomplete-information negotiation, after 16 target games, its estimated gain was .029 AUC when the forecaster knew the target’s valuation and .011 with public information alone.”