- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes an agent/methodology called LatentBot, which uses an opponent LLM's internal activations to learn strategic action decisions. The methodology is as follows: The history of rounds and the current round's prompt are fed to the opponent LLM. Then, before the opponent LLM outputs, LatentBot extracts the activations of the opponent LLM from a certain intermediate layer. It trains a lightweight classifier to predict the opponent's action from the latent activations. The experiments show that LatentBot has highly desirable performance in predicting the opponent's actions.
Critical strengths
-
Using activation probes as an input to another agent’s decisions moves probing beyond offline analysis and raises useful questions about communication between neural agents.
-
Methodologies such as training a regression model between different LLM activations to transfer activations, probing the activations using lightweight models and their success can be useful and transferable to the interpretability of LLMs.
Critical weaknesses
-
The activation-access assumption makes the prediction task substantially easier and limits applicability. LatentBot reads the opponent’s layer-14 activation at the final prompt position, directly accessing an intermediate computation that produces the opponent’s action. Predicting that action is therefore substantially easier than inferring an unknown strategy from observable interactions. Moreover, the method requires access to the opponent’s model and execution context, restricting its applicability. Also, since the activations need to be captured basically right before the output. I don't see the practicality of extracting the activations right before the output instead of simulating the LLM directly.
-
The extraction-layer choice is insufficiently justified. Although the transfer experiment mentions validation-based layer selection, the native and surrogate settings lack a documented selection procedure and a layer-wise sensitivity analysis.
-
The counterfactual reaction prediction simulates the current round and the next round. It has one step of lookahead. In a more complex game setting, a game tree might need to be gone through to search the best direction given the paper's methodology, which can be time consuming. The authors can consider how the agent should explore/learn the best outcome in a more complex setup.
-
It seems the black-box experiment assumes substantial knowledge of the opponent and does not establish recovery of its internal strategy.
Questions for authors
Q1: Can the authors justify the d