- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper studies whether requirements for memory use can be read from the activations of a frozen language model and translated into better control of generation. It proposes DUET, which predicts a control coefficient for each memory from a labeled reference bank, updates directions derived from activation differences during decoding, and adapts their strengths using changes in the output distribution. Experiments across six backbones and three benchmarks report improvements in selective memory use and joint satisfaction of multiple requirements.
Critical strengths
-
The paper addresses a meaningful problem and distinguishes reading strategies for memory use from enforcing them during generation. The limitations of fixed interventions motivate DUET, which reports substantial improvements on key metrics of memory use across multiple models.
-
The ablations and supporting analyses are relatively thorough. Ablations across component combinations and steering strengths, experiments varying calibration data size, and tests with references from an independent source help explain the contributions of the components and reference data. The results reveal both complementary effects on controlling multiple memories and trade-offs on individual metrics.
Critical weaknesses
-
The main comparisons use different test sets without explaining why. Appendix B.3 reports 252 baseline versus 285 DUET queries on RPEval, and 351 versus 370 query groups on BenchPreS, before invalid judgments are excluded. The reported gaps therefore cannot be cleanly attributed to the methods rather than differences in sample composition. All methods should be compared on the same test set, separate from calibration data.
-
The replacement judge lacks reliability validation. DUET uses a single 27B judge, whereas RPEval and SteeM use judges such as GPT-4.1 and Gemini-2.5-Pro, supported by comparisons with human judgments or other judges. This reliability evidence does not automatically transfer to the replacement, particularly for Support versus Dominate and graded memory dependence. Evaluation with the original benchmark judges, or capable independent judges larger than the 27B model used here, is needed to check whether the gains and rankings are robust.
Questions for authors
-
Why do the baseline and DUET test sets differ, and how do they overlap? Evaluating DUET and all baselines on the same test set for each benchmark would help confirm whether the reported gains hold.
-
Could the authors validate the key findings using the original benchmark judges or comparably capable larger models? This would help determine whether the conclusions remain robust to the choice of judge, given the smaller judge used here and the absence of human validation for the replacement.
5 comments
Response to Reviewer PF37 (2/2)
W2/Q2. Judge reliability and independent re-evaluation
Thank you for raising judge reliability. We selected Qwen3.8-27B for its overall capability, controlled local deployment, and evaluation reproducibility. Previously published Artificial Analysis Intelligence Index (version 4.3.2) results are as follows (higher is better):
| Model | AA Intelligence Index |
|---|---|
| Qwen3.8-27B (xhigh) | 34 |
| GPT-5.6 Sol (low) | 33 |
| Gemini 2.5 Pro | 16 |
| GPT-4.1 | 13 (AA estimate) |
These results place the reasoning variant of Qwen3.8-27B at a capability level comparable to some newer proprietary models and clearly above the earlier models listed above.
As described in Appendix B.3, we deploy the model locally with SGLang, enable deterministic inference, and set temperature to zero to improve evaluation stability and reproducibility. We also manually inspect a sample of evaluation results; 90% of the inspected judgments are considered reasonable.
We further re-evaluate the same responses with the same rubrics using the original benchmark judges, Sol, and Terra.
For RPEval, we re-evaluate all six methods on the same 285 Qwen3-4B queries. The table reports DUET's advantage over the strongest baseline; Few-shot is the strongest baseline for both metrics under all four judges.
| Judge | Multi gap (percentage points) | Mixed gap (percentage points) |
|---|---|---|
| Paper judge (27B) | +19.58 | +12.94 |
| GPT-4.1 (benchmark judge) | +10.49 | +0.00 |
| Sol | +28.67 | +25.88 |
| Terra | +23.08 | +15.29 |
Manual inspection identifies errors in GPT-4.1's interpretation of per-memory target labels. For example, a Few-shot response to query rpval_impl_multi_0022 has target AABC: Ignore the first two memories, Support the third, and let the fourth Dominate. The response recommends gentle bedtime stories according to the third memory while avoiding meditation, which the fourth explicitly rejects. GPT-4.1 instead interprets the first three memories as Ignore and penalizes the response, assigning 2/4. Sol and Terra correctly interpret the third requirement and assign 4/4.
We further randomly sample 10 Mixed queries and inspect one Few-shot and one DUET rationale per query, yielding 20 rationales per judge. Explicit label-reading errors are:
| Judge | Rationales with label-reading errors | Sample error rate |
|---|---|---|
| GPT-4.1 | 10/20 | 50.0% |
| Sol | 1/20 | 5.0% |
| Terra | 2/20 | 10.0% |
Mixed measures joint satisfaction of all memory-use requirements, so a local label-reading error can affect the entire query's judgment. Sol and Terra exhibit fewer such errors while retaining clear Multi and Mixed advantages, supporting DUET's multi-memory control effectiveness.
For SteeM, we add Gemini 2.5 Pro ratings on the previous random 30 groups, using the same 150 responses per method:
| Method | Gemini 2.5 Pro MAE ↓ |
|---|---|
| Few-shot | 1.293 |
| CoT | 1.207 |
| DUET | 1.200 |
Gemini's mean ratings of the same DUET responses by requested dependence level are:
| Judge | Target 1 | Target 2 | Target 3 | Target 4 | Target 5 |
|---|---|---|---|---|---|
| Gemini 2.5 Pro | 1.90 | 1.53 | 3.20 | 3.13 | 3.43 |
Gemini assigns mean ratings of 3.20, 3.13, and 3.43 to responses requested at levels 3, 4, and 5, without clear separation across these adjacent levels. Manual inspection also confirms fine-grained dependence-level confusions in some Gemini judgments and supports the evaluations of the paper's 27B judge, Sol, and Terra on the reviewed examples.
For example, a feedback-revision response about work-life boundaries cites historical details about boundary enforcement, communication scripts, and recovery activities. Gemini consequently assigns the highest level, 5. However, the response retains a self-contained explanatory structure and does not meet the highest-level requirement of directly continuing the history in a way whose full understanding depends on that specific history. Sol and Terra both assign 3, a judgment supported by manual inspection.
We then sample 30 of the 77 groups where DUET outperforms both Few-shot and CoT under the original 27B ratings, retaining all five target levels per group, and re-evaluate the same responses with the same rubric using Sol and Terra. The table reports baseline MAE minus DUET MAE; positive values favor DUET.
| Judge | MAE reduction vs Few-shot | MAE reduction vs CoT |
|---|---|---|
| Paper judge (27B) | +0.407 | +0.493 |
| Sol | +0.360 | +0.493 |
| Terra | +0.447 | +0.413 |
Both Sol and Terra reproduce the performance advantages on these 30 groups. We will include the cross-judge results, manually checked examples, and audit statistics in the final manuscript to clarify the basis of our evaluation.
We hope these results address your concerns and would be grateful if you could reconsider your rating.
Dear Reviewer PF37,
Thank you again for your thoughtful comments and constructive suggestions throughout the discussion. As the author–reviewer discussion period is coming to an end, we would like to kindly ask whether our responses have addressed your main concerns. If there are still aspects that remain unclear or require further clarification, we would be very happy to discuss them further.
We sincerely appreciate your time and valuable feedback.
Response to Reviewer PF37 (1/2)
We thank the reviewer for recognizing the importance of the problem, the distinction between strategy readout and execution, and our supporting analyses.
Due to time constraints before submission, we were unable to complete all methods on a common evaluation list. The earlier baseline and later DUET experiments used different calibration splits, resulting in different evaluation sets. We sincerely apologize for this inconsistency.
We have now completed all six methods across six backbones on common evaluation sets. The original overlaps and corrected evaluation populations are:
Current calibration data are excluded from the evaluation lists; few-shot examples come only from the corresponding calibration sets. Each metric is compared with its strongest baseline. Arrows denote baseline→DUET; differences are DUET minus baseline, and brackets show 95% paired intervals.
RPEval: joint satisfaction of multiple memory requirements. Multi measures joint satisfaction of all memory requirements; Mixed is the subset with mixed usage requirements. Higher is better.
BenchPreS: appropriate adoption, misapplication, and task completeness. AAR measures appropriate adoption, MR misapplication, and TC task completeness. Higher AAR/TC and lower MR are better. Differences are in percentage points for AAR/MR and on the original scale for TC.
SteeM: error in controlling memory dependence. Lower MAE is better.
On the common evaluation sets, DUET retains higher RPEval Multi accuracy, lower BenchPreS misapplication, and lower SteeM MAE than the strongest respective baselines on all six backbones; the paired 95% intervals exclude zero. Thus, DUET's core advantages in multi-memory control, reducing memory misuse, and regulating reliance remain.
We will update the main results tables in the final manuscript with these common-set results and revise the splits and experimental settings in Appendix B.3. We again apologize for the original inconsistency and thank you for bringing this issue to our attention.