acceptodds
Under review as a conference paper at ICLR 2027

Retrieving by Frequency, Not Position: Frequency-Division Multiplexed Memory for Structured Retrieval in Language Models

Abstract

We introduce a frequency-division multiplexed (FDM) context-encoded token stream with functional natural language coexistence in transformer models, wherein multiple independent information channels for structured data are encoded as amplitude-shift keyed (ASK) sinusoidal carriers, superposed into a single token sequence, enabling accurate, uniform forward-pass retrieval of in-context memory. Each fact targeted for storage and accurate retrieval access is a KEY=VALUE pair encoded into K discrete-valued channels as ASK sinusoidal carriers superposed as a composite signal into a fixed token sequence. Every token therefore carries information about every channel, making retrieval frequency-addressed rather than position addressed, without explicit Fourier, demodulation, or retrieval-specific modules. We finetune four transformer architectures (GPT-2 Medium, Qwen3-0.6B, LFM2.5-1.2B, Hermes3-3B) on up to 215K training samples of structured, non-linguistic FDM encoded data, achieving 93–100% per-channel retrieval accuracy across a 32-channel Context block (ch8–39). We then test whether the learned decoding is tied to a single position range. A brief additional finetuning stage teaches the FDM-finetuned model to decode the block at a second position range, and a learned key-value (K/V) prefix placed in front of the block does not degrade retrieval. We then finetune under a five-stage curriculum of decreasing carrier amplitude contrast, interleaving general-text replay where ordinary English text is mixed into every batch (annealed from 30% to 10%) specifically to guard the model from losing its natural-language abilities while it learns to read the FDM-encoded channels. This is performed on a 10-channel mixed-domain configuration, wherein all four architectures sustain 96.9–97.5% FDM action accuracy and 99.9–100% per-channel retrieval while answering 8–10 of 10 held-out natural-language questions correctly (10/10 for Qwen3-0.6B), with the same forward pass and no routing or gating mechanism handling both input types. These results show that transformer attention can learn to retrieve from a frequency-encoded in-context memory substrate while basic natural-language question answering remains functional. These results suggest a new design space for structured, frequency-addressed memory substrates that support short, accurate retrieval of structured facts within a single forward pass.

Primary area:
foundation or frontier models, including LLMs
Keywords:
frequency-division multiplexing, in-context memory, structured retrieval, attention mechanisms, key-value cache, amplitude-shift keying, transformer memory substrates, multi-hop reasoning, memory-augmented language models, positional bias

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_dteTOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes frequency-division multiplexed (FDM) memory: each of 40 categorical KEY=VALUE facts is encoded as an amplitude-shift-keyed sinusoidal carrier, and the carriers are summed, sampled at 256 points, quantized and mapped to vocabulary tokens, with an interleaved copy appended to form a 512-token block placed in the prompt. Four small models (GPT-2 Medium, Qwen3-0.6B, LFM2.5-1.2B, Hermes3-3B) are fully fine-tuned on 115K synthetic single-block examples and then 100K hop-structured examples to answer templated questions and write out all 32 context channels as NAME=VALUE pairs. The paper reports 93–100% per-channel accuracy, near-ceiling FACT accuracy up to 4 hops (82.8–83.4% at 5 hops), decoding at a second position range after additional fine-tuning, tolerance to a learned K/V prefix, and, with general-text replay, 8–10 of 10 held-out natural-language questions answered correctly, and concludes that attention can learn to retrieve from a frequency-addressed in-context memory.

Critical strengths

S1: The schema and templates are specified, and much of the code is released, so several claims can be checked.

The 40-channel schema (Table 8) and the n-hop question templates (Table 9) are listed in the appendix. The linked repository contains data generators, evaluation scripts and result files. Single-block training code is withheld, and the coexistence experiment of Table 7 is not included. This made it possible to verify several claims directly; as W3 and W4 show, however, that verification mostly undermines the paper's conclusions.

S2: The decoding results are consistent across four architectures.

GPT-2, Qwen3, LFM2.5 (a hybrid convolution–attention model) and Hermes3 all learn to decode the synthetic signal. Per-channel accuracy is high everywhere except LFM2.5's last channels (see W3). The prefix-tolerance experiment (Table 6) uses multiple seeds and reports Wilson intervals. These are the most carefully reported results in the paper, but they do not address the missing baselines.

Critical weaknesses

W1: The submission appears to violate double-blind review.

The Reproducibility Statement (p. 10, l. 486–489) links to a GitHub repository under a personal, non-anonymized account, which may reveal the author's identity. Anonymous code hosting was available and should have been used. I leave the policy decision to the AC.

W2: The presentation is poor, and there is not a single explanatory figure.

The paper contains no figures at all. The full pipeline (channel values → ASK carriers → composite signal → quantization → token mapping → interleaving → prompt and output format) is described only in prose and two equations, which makes the method hard to follow. There is also no Related Work section.

Several key details are missing:

  • The number of quantization levels L is not given in the method section; it can only be inferred from "log2 64" in §3.4.
  • The single-block question templates are never described. According to the released code, there are three: proceed, risk assessment, and share the secret.
  • The 10-channel mixed-domain configuration is not defined.

The text also contains inconsistencies:

  • Table 4 reports only three of the four architectures and omits Qwen3 without explanation.
  • The claim that joint accuracy "remains at 100% through 4 channels for all three architectures" (l. 297) conflicts with GPT-2's 99.8% at N=4 in the same table.
  • The Discussion refers to "up to three architectures tested (Table 2)", although Table 2 has four.
  • Hermes3's EXTRA accuracy is 99.8% in Table 2, but the note to Table 4 reports a single error on ch37 in 1,500 samples, which implies about 99.998%.

W3: The released code shows signs of insufficient verification and does not match the method and conclusions in the paper.

Signs of unverified generated code. The AI Use Statement says that LLM tools were used for "organization and editing of manuscript drafts" and that "Python experimental scripts were generated by these tools as were the evaluation of results of running these scripts". The released code shows signs that it was not carefully checked:

  • The LFM2.5 data-generation script (scaling_replication/eidetic_lfm2_fdm_e2e.py) loads its tokenizer from a non-existent model ID, "Qwen/LiquidAI LFM2.5-1.2B-Base". The Hermes3 script likewise uses "Qwen/Hermes-3-Llama-3.2-3B".
  • The GPT-2 "plaintext baseline" result file in nhop_composition/results/ is byte-identical to the GPT-2 FDM result file.

Encoding. Eq. (1) assigns each channel one of M amplitude levels over the whole signal. Every encoder in the released code, including the one in scaling_replication/ whose released per-model results match Table 3, instead does something different: it converts each value into bits and modulates each bit with binary ASK in a separate time segment of the signal. Part of the information is therefore carried by position, which is in tension with the title's claim of "frequency, not position".

Token mapping. The paper describes a fixed bijection τ from quantization levels to tokens. The code instead decodes the token IDs into a string and re-tokenizes it inside the prompt; the paper mentions re-tokenization only in passing in §3.2. Using the released encoder and tokenizers, I found that this round trip changes the token sequence in most samples: 182 of 300 for Qwen3, 282 of 300 for LFM2.5 and all 300 for Hermes3, with lengths of 510–548 tokens. In general, then, the model does not receive τ(q(x)).

Scoring. The paper says EXTRA requires the full NAME=VALUE string to appear verbatim. The scripts that reproduce Tables 2–4 (scaling_replication/eval_*.py, frequency_gradient/perchannel_eval_*.py, joint_retrieval/verified_repro/*/eval.py) only check whether the bare value appears anywhere in the response. Only the n-hop script checks NAME=VALUE.

High-frequency errors. The paper states that generation is capped at 250 new tokens (§2.1.3). It also notes that the high-frequency channels are the last ones in the Context block (§3.1.1). It does not test whether this explains the errors. With the released LFM2.5 tokenizer, every reference answer is 256–289 tokens long, so even a perfect model is cut off within the last channels.

Under the released scorer, truncation alone predicts the following, close to the reported values:

ch35–39EXTRAJoint accuracy, N=32
Truncated perfect modelabout 99 / 88 / 46 / 24 / 17%about 93%about 2%
Reported98.9 / 86.1 / 47.5 / 27.3 / 15.9%93.0%2.1%

The "high-frequency carrier degradation" on LFM2.5 is therefore most likely a truncation artifact.

W4: The experiments are toy-scale and lack the necessary baselines.

Task. The task is a fixed synthetic schema of 40 categorical facts with 2–5 values each, about 75 bits in total, answered through fixed templates. Single-key retrieval is tested only through four fixed 1-hop templates (STATUS, THREAT, RULE, LOCATION). The 32 context channels are never queried individually; they are always transcribed as a whole block in a fixed order.

Missing baselines.

  • There is no baseline that places the same facts in context as plain text. Written as NAME=VALUE lines, the 40 facts take roughly 220–310 tokens with these tokenizers, fewer than the 512-token FDM block.
  • There is no comparison with learned memory or context compression (gist tokens, ICAE, prefix-tuning).
  • Apart from the 25% RULE chance level in the Table 7 caption, no chance-level or ceiling reference is given. For example, in the released scorer, the FACT check for the single-block "Should {AGENT} proceed with the mission?" template passes whenever the agent name, which appears in the question, is copied into the answer.
  • Natural-language competence is judged from only 10 questions, with no measurement before fine-tuning, as the paper acknowledges.

Unreported plaintext baseline. The linked repository contains a matched plaintext baseline that the paper does not report, and its results are mixed. In the clean condition (corruption_experiments/README.md, 150 samples per condition):

ModelPlaintext context accuracyFDM context accuracy
Qwen3100%82.0%
LFM2.591.0%99.6%

FDM is ahead on both models only under stride deletion. The plaintext-baseline README (nhop_composition/plaintext_baseline/README.md) concludes that "under clean memory conditions, plaintext and FDM achieve comparable compositional accuracy when given matching compositional training". It argues that FDM's benefit lies in robustness under stress, which the paper does not evaluate.

The 5-hop drop. The n-hop plaintext baseline's 5-hop FACT score on Qwen3 is 4022/4825 (83.36%), exactly the same as FDM on Qwen3. With the released templates and scorer, even a perfect output reaches only about 83% at 5 hops, because the reference answer of the META-chain template omits some of the consulted values. The reported 5-hop drop is therefore a scoring artifact rather than a limit of FDM.

Questions for authors

Q1: Why is the plaintext baseline in the repository not reported, and in what setting does FDM outperform plain text?

This is a requested experiment that could change my assessment. Please report the plaintext baseline next to every FDM result, under the same training budget and generation limit, together with the token cost per fact. Please also include the corruption conditions in which FDM is claimed to help, and add learned-memory baselines (e.g., gist tokens, ICAE, prefix-tuning). Without them, the paper does not show a benefit of FDM over plain text.

Q2: Which encoder produced the reported results: the M-ary ASK of Eq. (1), or the time-segmented binary ASK in the released code?

This is a clarification request. If it is the time-segmented version, please explain how the "frequency, not position" claim still holds. Please also state how the decode-and-re-tokenize step mentioned in §3.2 changes the sequence the model actually sees.

Q3: Are the LFM2.5 high-channel errors caused by the 250-token generation limit?

This is a requested experiment. Please re-run Tables 2–4 with a substantially larger generation limit and with the Context block emitted in a shuffled channel order.

Q4: What are the chance-level and maximum achievable scores for each metric?

This is a clarification request. Please give chance and ceiling scores for each metric beyond the 25% given for RULE. Please also report the 5-hop FACT accuracy when only values present in the reference answer are scored, and explain the inconsistencies noted in W2. Finally, please state what is trained for the "4K steps" per seed in Table 6.

10 comments
AuthorsOpenReview ↗

Author response: method corrections (Eq. 1, tokenization, scoring), baselines, and revised PDF

We thank the reviewer for the very helpful comments and thorough review. The authors have rewritten the Introduction and Abstract for clarity of the technique, which comes from a field outside machine learning, and have added four figures in consideration of the reviewers’ comments about a lack of clarity. In particular Figure 4 outlines the full design and flow from channel values → ASK carriers → composite signal → quantization → token mapping → interleaving → prompt and output format. The prose, adjusted particularly with regard to Equation 1, is now supported by the added figures. The number of quantization levels is 64 and we have indicated it in the prose and in Figure 4’s caption. The single-block question templates are listed with the full channel definitions in Appendix A. In the interest of space and to contribute to the clarity of the paper’s main point, the 10-channel mixed-domain configuration has been removed. We thank the reviewer for pointing out the missing Qwen3 result in Table 4 and have added “This experiment was not conducted on the Qwen3 model and is not reported here”; it was not conducted for lack of resources and time. “Remains at 100%” has been adjusted to “remains at near 100%”. Hermes3’s EXTRA accuracy is corrected to 100.0%; the 99.8% was a transcription error (every Hermes3 run gives one error in 1,500 samples, on ch37). The reviewer is correct, and we thank them for checking the code. The three scaling_replication/eidetic_* scripts were templated from the Qwen3 script and the MODEL_ID strings (and the architecture notes in their docstrings) were never updated. This has been corrected in the repository. The data used in the paper was produced by the generators in joint_retrieval/verified_repro/{lfm2,hermes3}/generator.py, which carry the correct IDs (LiquidAI/LFM2.5-1.2B-Base, NousResearch/Hermes-3-Llama-3.2-3B); the MODEL_ID constant is used only for tokenizer loading in generate, and the token-map bijection depends only on VOCAB_SIZE and the seed, which were correct, so no reported result is affected. The eval paths load the finetuned checkpoints by path and were unaffected. The reviewer is also correct that the GPT-2 plaintext result file is a copy of the GPT-2 FDM n-hop result; the GPT-2 plaintext baseline was never completed, and the file has been removed.

The reviewer is correct about the released encoder (Q2). Equation (1) in the submission does not describe what the code does. The encoder writes each value index as B=3 bits, divides the 256-sample signal into three symbol intervals of 85 samples, and in each interval keys channel k’s carrier to A_high or A_low according to the corresponding bit. In this way, bit 1 of every fact lives in the first 85 samples, bit 2 in the next 85, bit 3 in the last 85. So once the system has found the right carrier, its amplitude level in those three places is read in order. Therefore this latter part is positional; the addressing of the fact is by frequency. Further, the 40 carriers are summed within every interval. This is binary ASK, bit-serial, and it is what produced the results presented in the original draft of the paper. We have corrected the revision accordingly: Equation (1) and §2.1 now state the bit-serial scheme explicitly, with the symbol interval T_b = 85/f_s; the captions of Figures 1, 2 and 4 describe the value as a three-bit on/off pattern across the intervals rather than a single amplitude; and the resolution argument in the Discussion has been corrected, since it used the full signal length N where the symbol interval N/B applies. Over one 85-sample interval the Fourier resolution is f_s/85 ≈ 1.18 Hz, which is above the 1 Hz carrier spacing, so the hosts’ 99–100% per-channel accuracy is not explained by carriers being resolvable within a single interval. The revision states this as an open question rather than a property of the design.

The reviewer is correct. The generator decodes the mapped token IDs to a string and the prompt is re-tokenized, so the host receives a re-tokenization of τ(q(x)), not τ(q(x)) itself. The string is determined by the levels and the tokenizer is deterministic, so the model’s input is still a fixed function of the signal; the model learned to read this re-tokenized version, and every number in the paper was measured on it. We have corrected §2.1 to state what the host actually receives, replaced “512 tokens” by “≈512” where an exact count was implied, and are checking whether the re-tokenization is lossless; the result will be stated in the revision.

The reviewer is correct. The scripts behind Tables 2–4 score EXTRA by checking whether the bare value appears anywhere in the response, and only the n-hop script checks the full NAME=VALUE string. The paper’s description of EXTRA is therefore wrong for those tables and has been corrected. Since several values are shared across channels (LOW/MEDIUM/HIGH, CLEAR, NONE, OPEN, CRITICAL, UNKNOWN), the bare-value check can credit a wrong channel when another channel carries the same value, so it can overstate accuracy. We are re-scoring the same 1,500 test samples per host with the strict NAME=VALUE check and will report the strict numbers in Tables 2–4 of the revision, with the lenient numbers alongside for comparison.

The reviewer’s truncation analysis is being tested directly (Q3): the same test samples are being re-run with a 512-token generation cap and scored at both caps. Pending that result, every reference to “high-frequency carrier degradation” has been removed from the revision and the LFM2.5 ch36–39 errors are described neutrally as errors in the last channels emitted.

W4. We accept the characterisation. The task is a fixed synthetic schema of 40 categorical facts with 2–5 values each, queried through fixed templates. As Appendix B shows, the FDM-encoded token sequence the model receives bears no surface relationship to the KEY=VALUE facts it encodes; to a reader it is a string of unrelated word fragments. The paper is a feasibility study of one question: can a transformer retrieve facts from such a sequence when the facts are addressed by carrier frequency rather than by position, at high accuracy and across architectures? On that question the results are positive and, as the reviewer’s own checks confirm, reproducible. The paper does not show, and the revision does not claim, that FDM is more accurate or more token-efficient than plain text on this task. The authors did not design the study to pit the technique against plaintext, and the rewritten Introduction and Abstract state the exact reason for having done it: to demonstrate that a single frequency-based technique, a simple modulation scheme known throughout a wide and mature communications-engineering field, can be used by four transformer language models for KEY=VALUE fact retrieval, which is relevant to the communication capabilities of language models and agents and to how information is distributed to humans.

The importance is in showing that the technique, rooted in decades of theory and laws outside machine learning and built on frequency for efficient transmission between transmitters and receivers, works and applies to language modelling, not that it is better than plaintext retrieval. The natural-language competence experiments have been removed from the paper, as they require a robust evaluation that has not been conducted.

On Q1 (why the plaintext baseline in the repository is not reported, and in what setting FDM outperforms plain text): the authors ran the plaintext comparison early in the experimental pipeline on Qwen3 and LFM2.5 (the GPT-2 run was never completed, as noted above), but the value of the technique is not as a replacement for plaintext retrieval in transformers, and we do not claim a setting in which it outperforms plain text. The result files are retained in the repository and the revision reports them in an appendix next to the FDM results, with the token cost of each representation. The FDM retrieval technique as designed is based on one mature frequency-based modulation technique, which offers a different way to organise tokens for language-model retrieval, without any FFT or demodulation module, resting on the idea of separable carriers, each carrying one fact, composed into one signal. Rather than compare to plaintext, the authors have attempted to demonstrate the extent of FDM as an accessible retrieval technique for transformers in context, and have begun to extend it toward the parametric regime: the revision introduces Table 6, a split-substrate experiment showing that KEY=VALUE facts injected into the K/V cache are read correctly while FDM-modulated KEY=VALUE facts in context are read accurately in the same forward pass.

We also agree with Q4 that META is not a retrieval measure as scored: 60% of samples are credited without reading the response, which is why all four hosts show 79.8–79.9%. This is stated in the caption of Table 2 and the column is not interpreted.

We greatly appreciate the thorough review by the reviewer of our work.

Reviewer_5x99OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces frequency addressed memory encoding that superposes 40 structured facts as sinusoidal waves and maps the quantized composite signal into 512 discrete tokens, distributing all information across the entire token sequence rather than specific positions. The paper fine-tune transformer architecture using a multi-stage curriculum and show that they can retrieve and compose target facts while retaining natural language capabilities.

Critical strengths

The authors show that standard transformers can learn to implicitly demodulate this non-linguistic composite signal and accurately retrieve target facts purely through next token prediction fine-tuning without any explicit signal processing modules or architectural changes.

Critical weaknesses
  1. Lack of high-level explanations makes understanding this paper very confusing. When adopting concepts outside ML, there should be considerable explanation and intuition about why specific approaches are adopted.

1.1. In Section 2, please explain why these experiment procedures is required for this system, i.e., “LLMs cannot directly process continuous waves (sin) as input since the can only accept discrete tokens. Therefore, we designed the following process to enable LLM deal with continuous signal.”

1.2. Lack of explanation on channels (Appendix A) and how input is constructed with FDM makes understanding this paper very difficult. Please at least include a section about channels and concrete model input/output in main paper rather than the appendix. The description in Section 2 is not enough. I suggest reducing the length of the abstract and using that space for more essential descriptions in the main text.

  1. For insisting that FDM retrieval coexists with natural language competence in Section 2.1.6 and Table 7, The paper evaluates the models on only 10 simple general knowledge questions without a pre-finetuning baseline. Evaluation on 10 questions is too weak to show that full parameter fine-tuning on over 100k non-linguistic sequences does not degrade general language abilities. How about evaluating the models on standard NLP benchmarks, e.g., MMLU, GSM-8K, before and after fine-tuning?

  2. Each channel only encodes a small and fixed vocab of 2 to 5 predefined values (Table 8). Meanwhile, encoding 40 such facts requires 512 tokens and over 215K fine-tuning samples. Setting 40 “Key=Value” pairs directly in plain text would use fewer tokens without requiring any fine-tuning (this may be a good baseline comparison). Practical advantage of spending 512 tokens to encode 40 small categorical variables compared to plain text remains unclear.

  3. It is unclear whether frequency itself matters. Would replacing the sinusoidal carriers with random orthogonal codes give different results?

Questions for authors

Please see section Critical Weaknesses.

3 comments
AuthorsOpenReview ↗

Author response: intuition for the pipeline and channels, withdrawn NL-competence claim, plaintext comparison, and the role of frequency

The authors would like to thank the reviewer for the thoughtful comments on this work and duly respond.

1.1. The authors agree that the sentence the reviewer quotes was unclear, and have rewritten it in the revision. What it was meant to say is this: a transformer accepts a sequence of discrete vocabulary tokens, not a continuous waveform. The composite signal x(t), the sum of the sinusoidal carriers that represent the separate frequency channels, therefore has to be sampled, quantized to L = 64 levels and mapped level-by-level onto a fixed subset of the vocabulary, so that the model receives the single composite signal as an ordinary token sequence, in the same way an analog-to-digital converter presents a waveform to a digital receiver. In other words, the communications stages in Figure 2 are not an analogy: sampling and quantization are the concrete steps that let a modulated signal enter a language model at all.

1.2. A channel here is simply a channel in the radio sense. One carrier frequency is assigned to each fact, the carriers are summed into one signal, and they remain separable in the spectrum of that signal. The authors have introduced three figures to address the lack of clarity on what channels mean and why this matters, in the context of mature communications engineering: Figure 1 shows the carriers and composite spectrum, Figure 2 the correspondence to a conventional communication system, and Figure 4 the concrete model input and output, from KEY=VALUE facts through carriers, sampling, quantization and token mapping to the prompt and the NAME=VALUE answer; the abstract has been shortened to make room. How separately retrievable the channels are rests on one simple law, the Nyquist limit: the sampling rate must be at least twice the highest carrier frequency for the signal to be fully decomposable. Respecting this law means that a fixed number of facts can be packed into one fixed token budget and retrieved by frequency without loss of data integrity.

The authors agree, and thank the reviewer. Ten questions cannot support the claim that full-parameter finetuning on 215K non-linguistic sequences preserves general language ability, and we have withdrawn the claim: Section 2.1.6 and Table 7 are removed from the revision, and the paper no longer asserts that FDM retrieval coexists with natural-language competence. The authors would like to complete a full evaluation in future work, using for example LAMBADA, HellaSwag, PIQA, ARC-Easy, ARC-Challenge, MMLU, WikiText (perplexity), GSM8K and IFEval, with identical settings before and after finetuning. The authors agree, and point to the plaintext experiments in the public repository linked in the paper, which the revision now reports. FDM does not improve clean accuracy or token efficiency over plaintext, and the revision does not claim that it does. We present it as an alternative to plaintext, in which a fact is addressed by its position in the token sequence and the model must locate it; with FDM, every fact is present in every token and is addressed by its carrier frequency. FDM is a well-known communications technique, and the fact that it is applicable to language models, as this early work shows, points to a long list of mature techniques from communications engineering that may be applicable as well. The authors are not claiming any inherent ability of a transformer model (the encoding and decoding require custom finetuning); we are pointing to the applicability of frequency-based engineering approaches that are well known, mature, backed by existing infrastructure, and that humans already use for high-efficiency communication between transmitters and receivers.

Frequency matters, relative to other encoding schemes, because it is what makes each fact separately addressable within the composite signal and what lets the same signal travel over a physical channel unchanged. The reviewer’s proposed ablation, replacing the sinusoidal carriers with random orthogonal codes, has not been run; it is a valid test of whether the frequency structure specifically, rather than orthogonality alone, is what the host exploits, and we note it in the revision as future work.

AuthorsOpenReview ↗

Plaintext comparison and carrier ablation now in the revision

The authors have uploaded a revised PDF. The plaintext comparison mentioned above is now reported in Appendix D (Table 11, beside the FDM results, with token cost in Table 10). The random-orthogonal-codes ablation the reviewer proposed is noted as future work in the Discussion.

Thank you to the authors for the response and manuscript revisions. However, my concerns regarding the method's practical utility and underlying mechanism remain unresolved.

W3: The added plaintext baseline (Appendix D) confirms that plain text achieves comparable accuracy using 1.7x to 2.3x fewer tokens, leaving the practical advantage of FDM unclear.

W4: The ablation with random orthogonal codes was deferred to future work, which is critical given the discussion with reviewer dteT showing that the single interval fourier resolution (1.18Hz) exceeds the 1Hz carrier spacing. Consequently, it remains unverified whether the model genuinely exploits frequency structure.

Reviewer_JF3zOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

It is hard to disentangle what problem this paper attempts to solve and the core contributions due to the unclear writing, but I would guess that this paper attempts to improve retrieval accuracy of LLMs, where 4 small sized LLMs are chosen in the experiments. The main contribution is the frequency-division multiplexed (FDM) encoding that uses amplitude-shift keying (ASK) for memory substrate design, but the paper never really defines what that is. From the methods section, it seems like a way to construct additional "memory" prompts from structured facts (e.g. Table 9) that can be added to a question input to an LLM that uses frequency-based terms.

Experiments are only on synthetic data and not the usual retrieval tasks such as needle-in-the-haystack or common datasets. The model sizes are also small, not exceeding 3B. No baselines are presented either and the results given are all similar accuracies across 4 models that incorporate their proposed encoding. There is a multi-hop retrieval task but it is not defined how the experiment is set up.

Critical strengths

None, because there is no baseline comparison or clear experimental goals. And all the retrieval accuracies presented seem similar across different models.

Critical weaknesses
  • The writing is extremely unclear. Even after reading the paper multiple times, I cannot understand both the method and experiments clearly. This is far below ICLR's bar and a major revision would be needed (therefore I would suggest a clear reject in this case).
  • Since the main problem it wants to tackle is retrieval, one would hope that (1) established retrieval benchmarks are used and (2) established/existing retrieval improvement methods are compared against. Neither is present in this paper and the method purely compares against itself in this paper.
Questions for authors
  • Could you provide results of your proposed methods on common retrieval benchmarks such as RULER?
  • Could you provide baselines as well to compare to? i.e. without the memory substrate/prompt?
4 comments
AuthorsOpenReview ↗

Author response: clarity revisions, RULER applicability, and plaintext/no-memory baselines

The authors would like to thank the reviewer for the thoughtful comments on this work and duly respond.

With regards to the comment on clarity, the authors have revised the submission accordingly. We have added three figures that ground the technique in standard communications engineering (Figures 1, 2 and 4: the frequency-domain view, the correspondence to a conventional communication system, and the encoding/decoding pipeline) and a fourth (Figure 3) that lays out the training stages and the experiments done in this work. We recognise that the paper asks the reader to follow one communications-engineering technique applied to language-model encoding, decoding and finetuning, and have therefore rewritten the Abstract and Introduction and simplified the text throughout. The theoretical description of the technique is retained, since it is now supported by the figures. The point of the technique is to pack separable facts into separate frequency channels of a single composite signal, place that signal in a fixed token budget, and have a transformer read it by ordinary inference, in such a way that we demonstrate a potentially meaningful capability for transformer models as they apply to retrieval, and potentially to communication between models and agents, that is well known in communication theory and widely implemented.

In response to the comment about the need for benchmarking: RULER measures retrieval of plaintext KEY=VALUE facts from long natural-language context. Our method however first encodes the facts using the traditional communications scheme of amplitude-shift keying (ASK) on frequency-division multiplexed (FDM) carriers. These FDM encoded facts essentially take 40 KEY=VALUE facts, assign a frequency to each of the 40, and turn them into one composite sinusoidal signal carrying 40 “channels” like radio. Each channel carries one KEY=VALUE fact, but the composite signal can be assigned a fixed transformer token budget. A large ASK encoded dataset is then used to finetune four models, to teach these to decompose up to 40 KEY=VALUE facts from one composite sinusoidal signal. The non-linguistic encoded sequence is shown in Appendix B. The single composite signal carrying 40 channels, each assigned to a frequency like a radio spectrum, is then assigned to one ≈512-token budget as one string of sequential tokens which is interpretable by a transformer language model. Running RULER as-is on our checkpoints would measure their natural-language retrieval, not their ability to decompose an ASK (FDM) encoded signal for retrieval; and RULER has no FDM condition.

The authors agree that baselines are needed. A matched plaintext baseline was run previously: same 40-channel memories, templates, train/val/test split and scoring, with the [MEMORY] block holding the facts as newline-separated NAME=VALUE pairs instead of FDM tokens, and the base model finetuned on the N-hop stage only. It reaches 100% Action at every hop count on Qwen3-0.6B and LFM2.5-1.2B, with Fact at 5 hops of 83.4/82.7%, identical to the FDM hosts; Qwen3 also reads all 32 context channels at 100%. FDM therefore does not improve clean accuracy over plaintext; its contribution is frequency addressing within a fixed token budget, as the revised Introduction states. To give the chance floor the reviewer asks for, we will add a no-memory control with the block removed, and the base models’ zero-shot read of the plaintext memory, on the same test split, within the time remaining to do so.

Reviewer_JF3zOpenReview ↗

Response to Rebuttal

Many thanks for your response and revising it to be clearer.

My concern with "the technique is to pack separable facts into separate frequency channels of a single composite signal, place that signal in a fixed token budget, and have a transformer read it by ordinary inference" is that this may not be generalizable across other datasets. These facts must be selected from some datasets and there is a limit to how much you can pack as you cannot pack every single related fact. This means that the method can only work if you fine-tune the LLM for every single dataset, which is cumbersome and even so you might even miss some facts (your K=40 is finite).

In order for the spectrum-based method to generalize, you somehow need to pack information into an abstract space and construct an adequate set of basis functions (e.g. like how you can represent functions in a RKHS and use the RKHS basis to construct approximations of regression functions).

Also, rather than packing them in FDM contexts, would it be better it just be better to build a knowledge database and then do fact retrieval instead? Then the decoding follows similarly (e.g. as commonly done in the HotpotQA benchmark) and it's also fast and doesn't require any fine-tuning (you could train it to leverage search engines too if you wanted to like Search-R1 (https://arxiv.org/pdf/2503.09516)). I suggest you also include these baselines otherwise it's difficult to judge whether the contribution is truly useful or not (right now there are no baselines).

AuthorsOpenReview ↗

Response to follow-up: generalization and the Nyquist ceiling, plaintext baseline (Appendix D), and retrieval-database comparison

The authors thank the reviewer for the follow-up comment, and agree with the requirement of baselines.

On generalization. The paper demonstrates feasibility with only K=40 channels, each with a small categorical vocabulary, with the host finetuned to read that encoding. It does not show that the technique transfers to another dataset without finetuning, and we do not claim that it does. The number 40, however, is not the limit, and we restricted it to that small number due to resources and budget constraints. Each fact, in this example, is assigned to one carrier at its own frequency, 1 Hz apart, and the composite signal is sampled at f_s = 100 Hz. One can sample at many rates besides the one we chose. By the Nyquist criterion a sampled signal represents only frequencies below f_s/2, for complete separability and signal recoverability; therefore at 1 Hz spacing at most 49 carriers can be separably placed, and K=40 was chosen to sit inside that ceiling. The context was 512 tokens. One can see what the scale is like when (1) the sampling rate is raised and (2) the token budget is raised with it: at the tested frequency resolution the ceiling is 3,199 carriers for a 32K context and 12,799 for 131K (Appendix E, Table 12), and about 195,000 for a 2M-token context. These are calculated values and have not been tested. On the reviewer’s suggestion: the sinusoidal carriers are already an orthogonal basis, a Fourier basis over the sampled interval; what is fixed in this work is what is placed on each basis function, a three-bit index into a known list of values. Generalising would mean modulating each carrier with a representation of the fact rather than an index, so that the basis carries an abstract space in the sense the reviewer describes. We find the RKHS framing a useful way to state that direction and note it as future work.

Regarding the lack of baselines, we have introduced the plaintext results of our experiments (Appendix D, Table 11). These were run on Qwen3-0.6B and LFM2.5-1.2B: the same 40-channel memories, templates, train/validation/test split and scoring as the hop-controlled experiments, with the facts in context as NAME=VALUE lines, which is the form in which a retrieval database would deliver them. That said, we would like to emphasize that we have never sought to present the technique as a competitor to plaintext, and this cuts back to the reviewer’s suggestion of using other retrieval methods as comparative baselines to the work and to FDM as a retrieval technique. The retrieval technique of a language model is not presented in this work as strictly a way of retrieving facts. It is one leg of a communication flow pipeline. If a language model can encode its outputs so that they are retrievable by a language model and by humans entirely in the spectral domain, then the breadth of technologies that a language model and a human have access to includes all that is contained within the traditional field of communications engineering. This end-to-end study of the applicability of FDM/ASK, the use of frequency to modulate in what is a plaintext/token domain, is a report on a potential basis for traditional communications between language models without traversing the network stack as decoded text; the transmit side and channel effects are not tested here and are future work, as is the comparison against retrieval agents such as Search-R1, which could not be trained and evaluated in the time remaining.

With the way the new draft is written, with the four explanatory figures that support the communications-theory narrative at the forefront, the value is in the frequency-domain encoding scheme, and the laws governing frequency-based communications therefore serve as the correct baseline context, if you will, for this paper.

AuthorsOpenReview ↗

Baselines added (Appendix F): hosts fall to chance without the FDM block; base models cannot do the task zero-shot

We thank the reviewer for the request for baselines, and have finally completed two controls, which have been added to Appendix F (Table 13). In the first, the FDM block is deleted from the prompt (an empty [MEMORY] block, same question, same finetuned model), so the model has nothing to read the facts from. With the memory block removed in this way, all four finetuned hosts fall to chance (strict EXTRA 25.0–27.5%, chance 27.1%; joint-32 0%), against 99.8–100% with the FDM block present, so the accuracy crucially comes from reading the block and not from anything memorised during finetuning.

In the second, the base models given the same facts as plaintext with no finetuning score 0–0.6% strict EXTRA on Qwen3, LFM2.5 and Hermes3; GPT-2 scores 33% by repeating the input lines. The finetuned plaintext baseline is in Appendix D (Table 11).

The first control therefore checks that the hosts are reading the block rather than reproducing answers memorised during finetuning; the second checks that the base models cannot do the task from plaintext without finetuning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.