acceptodds
Under review as a conference paper at ICLR 2027

Learning from Recovery: Localized Episodic Trajectory Memory for Closed-Loop Multiclass Decisions under Binary Feedback

Abstract

We study multiclass decision making when an agent acts before receiving feedback and an error reveals only that the emitted action was wrong, not the hidden correct class. Our primary question is whether a persistent memory bank can turn such binary-feedback experience into better future decisions and more effective bounded retries. We introduce Localized Episodic Trajectory Memory (LETM), which stores context-conditioned winner-over-rejected relations from successful recovery trajectories such as a1- -> a2- -> a3+. The recurrent representation core is frozen; the trajectory-learning path uses no BP/BPTT, autograd-based update, or optimizer. Two evidence layers support the claim. First, mechanism controls show that disabling or scrambling persistent memory sharply degrades both first-decision and five-attempt performance, while random no-repeat retries recover only part of the closed-loop benefit. Second, in a frozen 10-seed closeout on UCI-HAR, N-MNIST, DVS-Gesture, and SHD, adding localized trajectory memory to the Full memory controller improves mean First@1 by +2.96, +2.16, +3.18, and +4.32 percentage points. The First gains are paired-significant on UCI-HAR, N-MNIST, and SHD; DVS-Gesture instead shows significant gains in five-attempt recovery and conditional-recovery AUC. Mean attempts decrease on all four tasks, and Local exceeds Full at all 20 predeclared dataset-by-exposure checkpoints. A Global-Pairwise control and representation diagnostics further show that reusable memory depends on contextual localization and retrieval geometry, not merely on storing class-pair counts.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
binary feedback, episodic memory, weak supervision, online learning, closed-loop decision making, backpropagation-free learning, representation geometry

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_pS5BOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces LETM for multiclass decisions where an agent receives only correct or incorrect feedback. After a successful recovery, LETM stores context-specific preferences between the successful action and previously rejected alternatives, allowing similar future instances to benefit from past mistakes without updating the encoder or using memory learning. Across different harness, adding LETM to an existing memory-equipped controller improves first-attempt accuracy and reduces the number of attempts on all four datasets. Controls suggest that these benefits depend on persistent, context-sensitive retrieval rather than simply avoiding repeated actions or accumulating global class-pair statistics.

Critical strengths
  1. LETM improves mean first-attempt accuracy by at most 4% and reduces mean attempts across four datasets, with first-attempt gains statistically significant on three datasets.

  2. Comparisons against disabled or scrambled memory, random retries, and global pairwise memory help isolate the contribution of persistent, context-specific recovery information.

Critical weaknesses
  1. The main comparisons isolate components of the authors’ controller, while external learning methods use different supervision or update regimes. Comparisons with bandit-learning and episodic-memory methods adapted to the same binary-feedback and retry budget would better establish competitiveness.

  2. The experiments cover classification tasks with only 6–20 classes, deterministic correctness feedback, and repeated guesses on an unchanged instance.

  3. The method uses dataset-specific addressing representations, and the diagnostics show substantial performance changes when these representations change. This leaves uncertainty about how reliably LETM works with a fixed, generally applicable representation.

Questions for authors

See weakness.

Reviewer_F1jcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at classification where the system only learns whether its answer was right or wrong, and gets up to five attempts on the same example. When a retry finally succeeds, the proposed method (LETM) remembers that the correct answer beat the wrong ones it tried, and stores this together with the example’s representation. On future similar examples, these memories nudge the system’s first guess. The encoder is frozen and the memory path uses no backpropagation. The authors first show that turning off or scrambling the existing memory system hurts performance a lot. They then show that adding LETM to that system improves first-try accuracy by about 2–4 points on four sensor and neuromorphic datasets, significantly on three of them, and slightly reduces the number of attempts needed. They also find that how well memory works depends heavily on the structure of the representation used to look it up.

Critical strengths

The experiments are run with real discipline. The authors fixed their main 10-seed comparison in advance and kept it apart from later exploratory results, rather than mixing in their best numbers. They report where results are weak, like the DVS first-try result, whose confidence interval crosses zero (Section 6.2, Table 3), instead of hiding it. The reported differences are consistent with the main results table. This makes it easy to see what the paper does and does not show. The representation finding is interesting in its own right. In one DVS representation, class information is still clearly present by a simple probe, but nearest-neighbor lookup fails and the memory system collapses. This is a nice illustration that “the information is there” and “memory can actually find the right past experience” are different things, and I think it applies beyond this paper.

Critical weaknesses

The baseline that all gains are measured against isn’t explained. The “Full” controller in Eq. 7 combines several scores (positive, veto, stable, candidate) with hand-picked weights. These are only named, never defined, and the same goes for the action-interaction matrix used in the settling step (Eq. 8). There’s also no explanation of how the weights and other constants were chosen, or how the frozen encoder was built. Without this I can’t judge or reproduce the baseline, so it’s hard to know what the 2–4 point gains mean. Some reference results also look off. On SHD, the linear and MLP baselines in Table 7 are very low (6.94% and 11.42%; chance is 5%), while Full gets 40% (Table 2), and I’d like the authors to explain this. It would also help to compare against at least one existing method for learning from right/wrong feedback, adapted to allow retries. It’s not shown that the paper’s key idea is what helps. Once a retry succeeds, the system knows the true class, and on UCI-HAR and N-MNIST almost every example eventually succeeds. So LETM may simply be remembering the correct label for nearby examples, which is basically nearest-neighbor learning. The new part, remembering which wrong answers were beaten, is never tested on its own. The Global-Pairwise comparison doesn’t answer this, because it removes the example-specific lookup as well. The key missing experiment is the same memory storing only the correct answer, ideally run on the same set of episodes. Similarly, the retry results should be compared against simply trying the next-best guesses in order, not just random guesses.

Questions for authors

How was the frozen encoder built? Was it ever trained with labels? Can you fully define the parts of the Full controller and explain how its weights and other constants were chosen, and on which data? Can you add a version of the memory that stores only the correct answer, and a “try the next-best guesses in order” retry baseline? How does LETM compare? Why are the linear and MLP baselines in Table 7 so low on SHD? Do they use the same inputs as Full? Appendix H says the trajectory code is a reconstruction of an earlier version. Were all main results re-run with the code described in the paper? What changes between the 10 seeds?

Reviewer_CZguOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper explores a multi-armed bandit setting where an agent is allowed to attempt a given problem multiple times, up to five in the paper's setting, with intermediate correctness feedback between attempts. In this setting, the author's suggest LETM, which stores previously trajectories of attempts to inform future decisions. Their approach does not make any updates to the parameters of any networks to improve the First@1 performance of a frozen agent on several benchmarks.

Critical strengths

None

Critical weaknesses
  • The paper's overall presentation is weak and difficult to follow. Overall, the writing is using many technical words that do not convey anything meaningful. The problem setting is not motivated clearly, and it would help to further contextualize it with existing problem formulations such as a multi-armed bandit setting. Perhaps it would also be beneficial to add descriptions or examples of the benchmarks used in the paper in the context of their described problem setting. Figures, tables, and plots lack of a sense of polish. For example, Figure 1 is full of information that is not referenced anywhere, and the nodes and arrows are not consistently representing the same type of objects. The font in Table 1 is larger than others for some reason, and Figure 2 is much too large for the amount of information conveyed within it. The method presented in section 4 also lacks clarity, and does not clearly identify the different baselines used in Table 1 and Table 2. Equation 7, which I understand to be a large component of the agent's decision-making process introduces several unmotivated components with no additional explanation. In addition, there are several concrete coefficients with arbitrary values which are not motivated either in this section. Algorithm 1 is introduced with no reference to it, once again making it unclear which baseline this is referring to. See questions for additional concrete examples concerning the overall weak presentation.

  • The paper does not make clear how it is positioned in the current literature, nor is it clear what problem it is solving. I am not sure that the multi-armed bandit setting they are proposing with intermediate correctness feedback is well-motivated or positioned within the broader context of the literature. It is also not clear that the benchmarks used are well-motivated either. For example, why is the N-MNIST dataset used instead of some standard variation of MNIST where an agent is allowed to make multiple attempts at a digit's class? Moreover, both N-MNIST and MNIST can be solved trivially in the zero-shot sense without multiple attempts. In addition, the paper compares with no existing baselines, once again making it difficult to assess the impact of this work relative to similar work.

Questions for authors
  • Where do all the different coefficients of equation 7 come from? Were they swept and can be considered hyper-parameters? What about all the other variables sc⋅s_c^{\cdot}? Are they outputs from some learned network or some other heuristic value?

  • What exactly is the difference between Full, Global, and Local? It's referenced several times in the experiments, but not properly explained int he methodology section.

  • What is a Paper-1 claim referenced in Lines 312 and 447?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.